[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488288577 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002362] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007371] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010007] pid_max: default: 32768 minimum: 301 [ 0.011124] LSM: Security Framework initializing [ 0.012062] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014061] *** VALIDATE selinux *** [ 0.022277] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026555] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029106] *** VALIDATE tmpfs *** [ 0.030431] *** VALIDATE proc *** [ 0.032018] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034249] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041726] debug: unmapping init [mem 0xffffffff8f459000-0xffffffff8f460fff] [ 0.043161] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044625] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052253] rcu: Hierarchical SRCU implementation. [ 0.054348] smp: Bringing up secondary CPUs ... [ 0.055537] x86: Booting SMP configuration: [ 0.056024] .... node #0, CPUs: #1 #2 #3 [ 0.060457] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.211282] node 0 deferred pages initialised in 145ms [ 0.213136] devtmpfs: initialized [ 0.214304] x86/mm: Memory block size: 128MB [ 0.217204] gcov: version magic: 0x41383552 [ 0.219350] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.220135] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.221346] pinctrl core: initialized pinctrl subsystem [ 0.222266] [ 0.222773] ************************************************************* [ 0.223028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.224030] ** ** [ 0.225021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.226020] ** ** [ 0.227017] ** This means that this kernel is built to expose internal ** [ 0.228018] ** IOMMU data structures, which may compromise security on ** [ 0.229018] ** your system. ** [ 0.230018] ** ** [ 0.231016] ** If you see this message and you are not debugging the ** [ 0.232017] ** kernel, report this immediately to your vendor! ** [ 0.233021] ** ** [ 0.234018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.235017] ************************************************************* [ 0.236621] NET: Registered protocol family 16 [ 0.237463] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.238084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.239082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.240544] cpuidle: using governor menu [ 0.244032] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.248037] PCI: Using configuration type 1 for base access [ 0.251263] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.262118] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.266038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.270047] cryptd: max_cpu_qlen set to 1000 [ 0.273238] ACPI: Added _OSI(Module Device) [ 0.275019] ACPI: Added _OSI(Processor Device) [ 0.276015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.278016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.283233] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.290366] ACPI: Interpreter enabled [ 0.292085] ACPI: PM: (supports S0 S3 S4 S5) [ 0.294013] ACPI: Using IOAPIC for interrupt routing [ 0.295107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.298486] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.309238] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.311102] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.314026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.317104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.322217] acpiphp: Slot [2] registered [ 0.324131] acpiphp: Slot [5] registered [ 0.325172] acpiphp: Slot [6] registered [ 0.326140] acpiphp: Slot [7] registered [ 0.328142] acpiphp: Slot [8] registered [ 0.329227] acpiphp: Slot [9] registered [ 0.331146] acpiphp: Slot [10] registered [ 0.332145] acpiphp: Slot [3] registered [ 0.334116] acpiphp: Slot [4] registered [ 0.336162] acpiphp: Slot [11] registered [ 0.338142] acpiphp: Slot [12] registered [ 0.339162] acpiphp: Slot [13] registered [ 0.341150] acpiphp: Slot [14] registered [ 0.343081] acpiphp: Slot [15] registered [ 0.344078] acpiphp: Slot [16] registered [ 0.345090] acpiphp: Slot [17] registered [ 0.346073] acpiphp: Slot [18] registered [ 0.347086] acpiphp: Slot [19] registered [ 0.349172] acpiphp: Slot [20] registered [ 0.350083] acpiphp: Slot [21] registered [ 0.351101] acpiphp: Slot [22] registered [ 0.353152] acpiphp: Slot [23] registered [ 0.355127] acpiphp: Slot [24] registered [ 0.356190] acpiphp: Slot [25] registered [ 0.358141] acpiphp: Slot [26] registered [ 0.360128] acpiphp: Slot [27] registered [ 0.361119] acpiphp: Slot [28] registered [ 0.363230] acpiphp: Slot [29] registered [ 0.365117] acpiphp: Slot [30] registered [ 0.366118] acpiphp: Slot [31] registered [ 0.368125] PCI host bridge to bus 0000:00 [ 0.369028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.371029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.374032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.376026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.379045] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.381032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.382181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.384963] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.387161] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.396021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.400727] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.402022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.404019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.407020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.408764] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.412729] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.414042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.417757] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.426022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.442023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.451017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.458813] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.472028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.482021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.501022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.511759] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.523018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.544022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.561016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.572550] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.581019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.591025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.616018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.627519] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.637017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.644031] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.665025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.680547] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.687016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.699018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.721017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.734000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.744030] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.752023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.772020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.783048] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.786439] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.789388] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.792424] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.795326] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.800104] iommu: Default domain type: Passthrough [ 0.802729] SCSI subsystem initialized [ 0.804237] ACPI: bus type USB registered [ 0.806204] usbcore: registered new interface driver usbfs [ 0.808103] usbcore: registered new interface driver hub [ 0.810102] usbcore: registered new device driver usb [ 0.812298] pps_core: LinuxPPS API ver. 1 registered [ 0.814030] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.818080] PTP clock support registered [ 0.820060] EDAC MC: Ver: 3.0.0 [ 0.822005] PCI: Using ACPI for IRQ routing [ 0.823838] NetLabel: Initializing [ 0.825014] NetLabel: domain hash size = 128 [ 0.827014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.829086] NetLabel: unlabeled traffic allowed by default [ 0.831143] vgaarb: loaded [ 0.832274] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.834023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.842000] clocksource: Switched to clocksource kvm-clock [ 0.948694] VFS: Disk quotas dquot_6.6.0 [ 0.950424] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.953793] *** VALIDATE ramfs *** [ 0.955382] *** VALIDATE hugetlbfs *** [ 0.957383] pnp: PnP ACPI init [ 0.960104] pnp: PnP ACPI: found 6 devices [ 0.983558] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.987686] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.990393] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.993075] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.996283] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.998813] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.002446] NET: Registered protocol family 2 [ 1.005112] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.010399] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.014438] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.019320] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.023268] TCP: Hash tables configured (established 65536 bind 65536) [ 1.026378] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.029572] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.032551] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.035762] NET: Registered protocol family 1 [ 1.039555] RPC: Registered named UNIX socket transport module. [ 1.041959] RPC: Registered udp transport module. [ 1.043855] RPC: Registered tcp transport module. [ 1.045068] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.047305] NET: Registered protocol family 44 [ 1.048633] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.050739] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.052686] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.054819] PCI: CLS 0 bytes, default 64 [ 1.056511] Unpacking initramfs... [ 2.473888] debug: unmapping init [mem 0xffff99d8fcc54000-0xffff99d8fffbffff] [ 2.478065] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.480321] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.483230] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.985551] Initialise system trusted keyrings [ 2.987668] Key type blacklist registered [ 2.992052] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.002404] zbud: loaded [ 3.004643] *** VALIDATE nfs *** [ 3.006008] *** VALIDATE nfs4 *** [ 3.008217] pstore: using deflate compression [ 3.012179] Platform Keyring initialized [ 3.128991] NET: Registered protocol family 38 [ 3.131207] Key type asymmetric registered [ 3.133330] Asymmetric key parser 'x509' registered [ 3.135091] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.138458] io scheduler mq-deadline registered [ 3.140920] io scheduler kyber registered [ 3.142985] io scheduler bfq registered [ 3.145149] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.148862] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.152616] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.156281] ACPI: Power Button [PWRF] [ 3.163582] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.170984] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.187180] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.198913] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.223725] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.253348] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.281412] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.286190] Non-volatile memory driver v1.3 [ 3.288099] Linux agpgart interface v0.103 [ 3.318139] virtio_blk virtio1: [vda] 146200 512-byte logical blocks (74.9 MB/71.4 MiB) [ 3.321741] vda: detected capacity change from 0 to 74854400 [ 3.337578] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.341097] vdb: detected capacity change from 0 to 1073741824 [ 3.360957] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.363907] vdc: detected capacity change from 0 to 2621440000 [ 3.383766] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.386465] vdd: detected capacity change from 0 to 2621440000 [ 3.403522] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.406953] vde: detected capacity change from 0 to 4294967296 [ 3.421600] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.425607] vdf: detected capacity change from 0 to 4294967296 [ 3.433131] libphy: Fixed MDIO Bus: probed [ 3.440568] usbcore: registered new interface driver usbserial_generic [ 3.443735] usbserial: USB Serial support registered for generic [ 3.446495] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.450758] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.452781] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.457567] mousedev: PS/2 mouse device common for all mice [ 3.464455] rtc_cmos 00:05: RTC can wake from S4 [ 3.468229] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.469906] rtc_cmos 00:05: registered as rtc0 [ 3.474664] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.477314] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.478623] intel_pstate: CPU model not supported [ 3.483884] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.488837] hid: raw HID events driver (C) Jiri Kosina [ 3.493082] usbcore: registered new interface driver usbhid [ 3.497382] usbhid: USB HID core driver [ 3.503374] drop_monitor: Initializing network drop monitor service [ 3.507456] Initializing XFRM netlink socket [ 3.510877] NET: Registered protocol family 10 [ 3.514682] Segment Routing with IPv6 [ 3.516743] NET: Registered protocol family 17 [ 3.518960] mpls_gso: MPLS GSO support [ 3.527316] RAS: Correctable Errors collector initialized. [ 3.529972] AVX version of gcm_enc/dec engaged. [ 3.531944] AES CTR mode by8 optimization enabled [ 3.618497] sched_clock: Marking stable (3618460974, 0)->(4526653226, -908192252) [ 3.622570] registered taskstats version 1 [ 3.625322] Loading compiled-in X.509 certificates [ 3.630453] zswap: loaded using pool lzo/zbud [ 3.657285] Key type big_key registered [ 3.670367] Key type encrypted registered [ 3.671858] ima: No TPM chip found, activating TPM-bypass! [ 3.674211] ima: Allocated hash algorithm: sha1 [ 3.676270] ima: No architecture policies found [ 3.678234] evm: Initialising EVM extended attributes: [ 3.680320] evm: security.selinux [ 3.681663] evm: security.ima [ 3.682871] evm: security.capability [ 3.684382] evm: HMAC attrs: 0x1 [ 3.686914] rtc_cmos 00:05: setting system clock to 2026-08-26 08:41:11 UTC (1787733671) [ 3.693902] debug: unmapping init [mem 0xffffffff90403000-0xffffffff905fffff] [ 3.697316] debug: unmapping init [mem 0xffffffff8f182000-0xffffffff8f458fff] [ 3.705155] Write protecting the kernel read-only data: 28672k [ 3.708263] debug: unmapping init [mem 0xffffffff8d803000-0xffffffff8d9fffff] [ 3.711668] debug: unmapping init [mem 0xffffffff8e114000-0xffffffff8e1fffff] [ 3.748888] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.758167] systemd[1]: Detected virtualization kvm. [ 3.760389] systemd[1]: Detected architecture x86-64. [ 3.762559] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.791454] systemd[1]: No hostname configured. [ 3.793513] systemd[1]: Set hostname to . [ 3.795888] random: systemd: uninitialized urandom read (16 bytes read) [ 3.798852] systemd[1]: Initializing machine ID from random generator. [ 3.842495] random: ln: uninitialized urandom read (6 bytes read) [ 3.932547] random: systemd: uninitialized urandom read (16 bytes read) [ 3.936108] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.941706] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.945958] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.695639] device-mapper: uevent: version 1.0.3 [ 4.698551] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.548860] virtio_net virtio0 ens2: renamed from eth0 [ 5.608986] random: fast init done [ 5.634562] scsi host0: ata_piix [ 5.642673] scsi host1: ata_piix [ 5.644360] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.647077] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.075181] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.189304] random: crng init done [ 10.190736] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.276142] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.862495] printk: systemd: 24 output lines suppressed due to ratelimiting [ 14.831263] SELinux: Disabled at runtime. [ 15.003093] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.026228] systemd[1]: Detected virtualization kvm. [ 15.029250] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.552502] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.557459] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.565562] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.571470] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.579056] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.609440] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.629430] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ 17.117962] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.041162] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.818891] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.872835] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.264800] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.289325] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 25.867566] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 26.350284] NFS: Registering the id_resolver key type [ 26.359703] Key type id_resolver registered [ 26.364433] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg319-server login: [ 76.430042] libcfs: loading out-of-tree module taints kernel. [ 76.515754] Key type ._llcrypt registered [ 76.517361] Key type .llcrypt registered [ 76.620494] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 92.465583] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 93.969587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 93.988640] alg: No test for adler32 (adler32-zlib) [ 95.476125] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 96.310526] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 98.071995] Key type lgssc registered [ 100.128746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 118.846342] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 158.557729] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 171.570242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 171.616613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.961265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.991887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 173.068297] Lustre: lustre-MDT0000: new disk, initializing [ 173.190908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.216464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.076455] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.590760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.784369] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 193.858751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 193.865533] Lustre: Skipped 1 previous similar message [ 194.061844] Lustre: lustre-MDT0001: new disk, initializing [ 194.193471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 194.246418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 194.261131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 198.917502] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.841436] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 213.813597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.158467] Lustre: lustre-OST0000: new disk, initializing [ 214.162933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 214.170864] Lustre: 8443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.251476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.594045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 215.603703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 215.655117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.303721] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.659908] hrtimer: interrupt took 13951901 ns [ 235.212076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.340817] Lustre: lustre-OST0001: new disk, initializing [ 235.344694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 235.350664] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.419124] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.935638] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.816121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 243.828097] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 243.872148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 254.071917] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.881297] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.966278] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing check_logdir /tmp/testlogs/ [ 271.792901] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing yml_node [ 276.291468] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 279.482452] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 282.053909] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 284.590630] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 04:45:50 EDT 2026 [ 302.148373] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 303.731658] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.997024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.001852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 321.013966] Lustre: Skipped 1 previous similar message [ 321.038300] Lustre: Skipped 3 previous similar messages [ 327.761418] Lustre: server umount lustre-MDT0000 complete [ 331.232521] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.252309] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 331.617171] LustreError: 7920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787733999 with bad export cookie 6958088809571485871 [ 331.617529] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 331.625049] LustreError: 7920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 336.354505] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.356375] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.358318] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.372788] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 336.412595] Lustre: Skipped 3 previous similar messages [ 338.032420] Lustre: server umount lustre-MDT0001 complete [ 342.266893] Lustre: server umount lustre-OST0000 complete [ 346.793537] Lustre: server umount lustre-OST0001 complete [ 353.964764] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 363.648451] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 398.914461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.103881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 399.145602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 399.225816] Lustre: lustre-MDT0000: new disk, initializing [ 399.336146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.360094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 403.528496] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 413.845755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.957991] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 414.009422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 414.015991] Lustre: Skipped 1 previous similar message [ 414.088866] Lustre: lustre-MDT0001: new disk, initializing [ 414.200739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 414.232039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 414.238669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.164450] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 427.919982] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 434.271184] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 440.608152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 440.877192] Lustre: lustre-OST0000: new disk, initializing [ 440.884357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 440.891970] Lustre: 17038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 440.990036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.521416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 442.531079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 442.636745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 446.557959] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 455.382282] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 459.797770] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 460.061372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 464.512977] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 464.713951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 468.455446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 468.463555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.467146] LustreError: Skipped 1 previous similar message [ 468.469966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 468.469976] Lustre: Skipped 1 previous similar message [ 468.513679] Lustre: Skipped 1 previous similar message [ 472.753748] Lustre: server umount lustre-OST0000 complete [ 478.688948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.703188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 478.708545] Lustre: Skipped 2 previous similar messages [ 482.867054] Lustre: server umount lustre-MDT0000 complete [ 483.809523] LustreError: 15077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.829021] LustreError: 15077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 486.684726] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734154 with bad export cookie 6958088809571487404 [ 486.692909] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.699033] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 487.072101] Lustre: server umount lustre-MDT0001 complete [ 492.247929] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:49:18 (1787734158) [ 509.844087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.470212] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.543988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.824973] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 515.558756] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.248290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.734436] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 534.633505] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 539.840196] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 541.924832] Lustre: Failing over lustre-MDT0000 [ 544.149324] Lustre: server umount lustre-MDT0000 complete [ 544.228209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 544.241064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.259969] Lustre: Skipped 1 previous similar message [ 544.266180] LustreError: 20644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.294488] LustreError: 20644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 562.592105] Lustre: 3638:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787734214/real 1787734214] req@ffff99d847aa9c00 x1874574719477248/t0(0) o400->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787734230 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 562.628489] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 562.647110] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.661656] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 563.250945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.908636] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x609018e822211432 [ 572.927210] Lustre: MGC192.168.203.119@tcp: Connection restored to 0@lo (at 0@lo) [ 577.759785] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 578.535431] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 578.546397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 578.575275] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 589.701254] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 599.018726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.022609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 599.022800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 599.028605] Lustre: Skipped 2 previous similar messages [ 600.493414] Lustre: server umount lustre-MDT0000 complete [ 604.128131] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.166696] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 604.624995] LustreError: 19896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734272 with bad export cookie 6958088809571488818 [ 604.634927] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 604.637540] LustreError: 19896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 605.086254] Lustre: server umount lustre-MDT0001 complete [ 613.891434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.571589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.584414] Lustre: Skipped 2 previous similar messages [ 618.432929] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 626.334465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.886570] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 638.568887] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 644.579951] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 646.429959] Lustre: Failing over lustre-MDT0000 [ 646.443471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.445735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 646.457904] Lustre: Skipped 1 previous similar message [ 646.473351] Lustre: Skipped 2 previous similar messages [ 646.630166] Lustre: server umount lustre-MDT0000 complete [ 664.714738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.844517] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 665.177702] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 668.868665] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 670.179080] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 670.192655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 670.197977] Lustre: Skipped 1 previous similar message [ 670.224883] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 677.541529] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 684.513084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.520259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 684.524261] Lustre: Skipped 1 previous similar message [ 687.279971] Lustre: server umount lustre-MDT0000 complete [ 689.633547] LustreError: 23250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.650082] LustreError: 23250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 690.720971] LustreError: 23237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734358 with bad export cookie 6958088809571489756 [ 690.726568] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 690.735822] LustreError: 23237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 690.995981] Lustre: server umount lustre-MDT0001 complete [ 698.292803] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:52:44 (1787734364) [ 712.639438] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 722.047726] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 758.132709] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 766.341674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 766.520310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 766.548231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 766.598084] Lustre: lustre-MDT0000: new disk, initializing [ 766.655372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 766.661770] Lustre: Skipped 2 previous similar messages [ 766.678187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 770.179785] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 779.004303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 779.090861] Lustre: 29628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 779.269418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 783.355387] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 794.697765] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 800.580860] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 806.953367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 807.142949] Lustre: 31593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 809.139506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 809.174445] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 809.228396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 809.238323] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 812.186629] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 820.015584] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 827.006639] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 845.281791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 845.282982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 845.291925] LustreError: Skipped 1 previous similar message [ 845.302174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 845.317136] Lustre: Skipped 2 previous similar messages [ 849.341825] Lustre: server umount lustre-OST0000 complete [ 858.653669] Lustre: server umount lustre-MDT0000 complete [ 860.643473] LustreError: 29633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.663692] LustreError: 29633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 861.781869] LustreError: 29620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734529 with bad export cookie 6958088809571490162 [ 861.798198] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 862.160841] Lustre: server umount lustre-MDT0001 complete [ 871.515886] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 874.006231] Key type lgssc unregistered [ 874.243526] LNet: 33573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.256147] LNetError: 33573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.273326] LNet: Removed LNI 192.168.203.119@tcp [ 874.985673] Key type .llcrypt unregistered [ 874.990085] Key type ._llcrypt unregistered [ 896.722280] Key type ._llcrypt registered [ 896.724113] Key type .llcrypt registered [ 896.810678] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 898.051378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 898.086474] alg: No test for adler32 (adler32-zlib) [ 899.167288] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 899.401087] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 901.063169] Key type lgssc registered [ 902.013897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 908.407720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 908.429045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.796909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.337535] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 920.037346] LustreError: 34987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.052070] LustreError: 34987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 920.145492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 920.382283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 924.062438] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 930.531554] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 934.545390] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 939.548077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 939.900280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.420661] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 946.039206] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 950.026950] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 966.658943] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:57:13 (1787734633) [ 968.651456] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 970.370758] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 976.448117] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:57:22 (1787734642) [ 983.753751] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:57:30 (1787734650) [ 984.626396] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 990.973496] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 990.978899] Lustre: Skipped 2 previous similar messages [ 992.457588] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:57:39 (1787734659) [ 995.909793] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 995.915184] Lustre: Skipped 3 previous similar messages [ 1016.210189] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1016.227552] Lustre: Skipped 6 previous similar messages [ 1018.637877] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:58:05 (1787734685) [ 1021.471765] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1025.440282] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1033.322401] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:58:20 (1787734700) [ 1034.991831] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1034.997867] Lustre: Skipped 3 previous similar messages [ 1046.836862] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:58:33 (1787734713) [ 1058.788738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1058.793549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.809710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1061.860550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.864932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1061.866235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1061.894648] Lustre: Skipped 1 previous similar message [ 1064.854674] Lustre: server umount lustre-OST0000 complete [ 1069.025038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1069.038920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.051904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.096837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.102880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.139674] Lustre: server umount lustre-MDT0000 complete [ 1077.216357] LustreError: 37477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.236130] LustreError: 37477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1079.049134] LustreError: 36829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734746 with bad export cookie 4998558531152535718 [ 1079.056338] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1079.077555] LustreError: 36829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1079.553293] Lustre: server umount lustre-MDT0001 complete [ 1093.146787] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1095.945514] Key type lgssc unregistered [ 1096.230710] LNet: 42832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.244143] LNetError: 42832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.276126] LNet: Removed LNI 192.168.203.119@tcp [ 1097.055911] Key type .llcrypt unregistered [ 1097.061671] Key type ._llcrypt unregistered [ 1116.317457] Key type ._llcrypt registered [ 1116.319965] Key type .llcrypt registered [ 1116.401345] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 1132.209051] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1133.152594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1133.169906] alg: No test for adler32 (adler32-zlib) [ 1134.238684] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 1134.425485] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1136.063189] Key type lgssc registered [ 1137.128792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1170.668744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1170.736211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.958207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1171.988353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1172.054990] Lustre: lustre-MDT0000: new disk, initializing [ 1172.116442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1172.130984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1176.741929] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1187.868124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.994622] Lustre: 45992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1188.059935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1188.064335] Lustre: Skipped 1 previous similar message [ 1188.158607] Lustre: lustre-MDT0001: new disk, initializing [ 1188.286823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1188.328551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1188.339898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1193.000290] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1203.278150] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1208.562951] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1215.538108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1215.761888] Lustre: lustre-OST0000: new disk, initializing [ 1215.765832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1215.775978] Lustre: 47974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1215.881336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.704491] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1222.221846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1222.236643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1222.375700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1231.186077] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1235.418118] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1235.620893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1240.116456] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1240.318143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1243.104903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1243.108398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.118845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1243.143132] Lustre: Skipped 1 previous similar message [ 1243.157420] Lustre: Skipped 1 previous similar message [ 1248.231960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1248.239967] Lustre: Skipped 1 previous similar message [ 1248.449492] Lustre: server umount lustre-OST0000 complete [ 1253.348630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.352111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1253.363898] Lustre: Skipped 1 previous similar message [ 1253.380166] Lustre: Skipped 1 previous similar message [ 1258.467444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.472771] Lustre: Skipped 1 previous similar message [ 1258.876577] Lustre: server umount lustre-MDT0000 complete [ 1262.994197] LustreError: 45984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734930 with bad export cookie 8939244404326083947 [ 1263.007937] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1263.013794] LustreError: 45984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1263.558469] Lustre: server umount lustre-MDT0001 complete [ 1272.522992] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:02:19 (1787734939) [ 1279.139290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.593201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.317671] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1285.089030] LustreError: 50437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1285.114282] LustreError: 50437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1290.212584] LustreError: 50438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.122539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.446957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1297.863154] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1304.370750] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1309.584931] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1316.401976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1316.770116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.537896] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1328.924704] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1333.250591] LustreError: 52972:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1334.170349] Lustre: Modifying parameter general.timeout=1 in log params [ 1335.173271] Lustre: Modifying parameter general.timeout=2 in log params [ 1337.213795] Lustre: Modifying parameter general.timeout=4 in log params [ 1337.219537] Lustre: Skipped 1 previous similar message [ 1342.026353] Lustre: Modifying parameter general.timeout=9 in log params [ 1342.035300] Lustre: Skipped 4 previous similar messages [ 1350.339609] Lustre: Modifying parameter general.timeout=19 in log params [ 1350.348029] Lustre: Skipped 9 previous similar messages [ 1367.043937] Lustre: Modifying parameter general.timeout=35 in log params [ 1367.048456] Lustre: Skipped 15 previous similar messages [ 1400.097597] Lustre: Modifying parameter general.timeout=65 in log params [ 1400.102755] Lustre: Skipped 29 previous similar messages [ 1470.287345] Lustre: Disabling parameter general.timeout= in log params [ 1470.294801] Lustre: Skipped 64 previous similar messages [ 1473.142745] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:05:39 (1787735139) [ 1476.156517] Lustre: 59922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1477.942613] Lustre: 60021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1481.576098] Lustre: 60233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1481.593422] Lustre: 60233:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1511.256206] Lustre: 61786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1511.266556] Lustre: 61786:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1511.317304] LustreError: 61786:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1515.593568] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:06:22 (1787735182) [ 1527.264651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.269633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.291481] Lustre: Skipped 2 previous similar messages [ 1527.303532] Lustre: Skipped 2 previous similar messages [ 1533.351804] Lustre: server umount lustre-MDT0000 complete [ 1536.737941] LustreError: 50418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735204 with bad export cookie 8939244404326084829 [ 1536.742782] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1536.752420] LustreError: 50418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1537.068096] Lustre: server umount lustre-MDT0001 complete [ 1550.813393] Lustre: server umount lustre-OST0000 complete [ 1574.472500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1574.940967] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1575.376164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.238551] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1588.401860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.546884] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1588.582894] Lustre: 64320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1594.480241] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1603.358922] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1609.968558] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1616.958421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1617.223627] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1617.529527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.548080] Lustre: Skipped 1 previous similar message [ 1623.687737] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1630.321328] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1632.094938] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1632.112514] Lustre: Skipped 23 previous similar messages [ 1632.181045] Lustre: 66884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1640.417554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1640.427233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.459637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1640.467066] Lustre: Skipped 3 previous similar messages [ 1645.020413] Lustre: server umount lustre-OST0000 complete [ 1650.147389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1650.157600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.166835] Lustre: Skipped 1 previous similar message [ 1655.571849] Lustre: server umount lustre-MDT0000 complete [ 1656.288871] LustreError: 65094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.307731] LustreError: 65094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1659.372080] LustreError: 64312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735327 with bad export cookie 8939244404326087979 [ 1659.379923] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1659.394968] LustreError: 64312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1659.738261] Lustre: server umount lustre-MDT0001 complete [ 1669.842351] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1672.184512] Key type lgssc unregistered [ 1672.468288] LNet: 68101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.475715] LNetError: 68101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.507605] LNet: Removed LNI 192.168.203.119@tcp [ 1673.247370] Key type .llcrypt unregistered [ 1673.250781] Key type ._llcrypt unregistered [ 1694.592463] Key type ._llcrypt registered [ 1694.595430] Key type .llcrypt registered [ 1694.694425] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1695.474566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1695.534178] alg: No test for adler32 (adler32-zlib) [ 1696.530374] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 1696.751384] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1698.383241] Key type lgssc registered [ 1699.435648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1706.204844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1706.231234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.066189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1712.724360] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1717.218785] LustreError: 69513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1717.242922] LustreError: 69513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1721.090861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.777670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1726.373849] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1733.508849] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1739.383422] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1746.809598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1747.537117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.340801] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1763.466950] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1775.377065] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1787.371786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.376104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1787.390974] Lustre: Skipped 2 previous similar messages [ 1787.407759] Lustre: Skipped 2 previous similar messages [ 1790.186206] Lustre: server umount lustre-MDT0000 complete [ 1792.480920] LustreError: 72012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1792.502280] LustreError: 72012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1793.255784] LustreError: 69495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735461 with bad export cookie 15969255508629130826 [ 1793.260022] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1793.264850] LustreError: 69495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1793.526946] Lustre: server umount lustre-MDT0001 complete [ 1807.438222] Lustre: server umount lustre-OST0000 complete [ 1832.445306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.753922] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1832.827735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1833.236306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1838.155166] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1848.985516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.175674] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1849.219479] Lustre: 74389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1849.856477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1853.835888] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1859.751134] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1864.892294] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1872.084793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1872.265356] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1872.994893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.208259] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1887.341207] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1888.931487] Lustre: 76963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1890.529047] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1898.981758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1898.982220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.983475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1899.015803] LustreError: Skipped 1 previous similar message [ 1900.839632] Lustre: server umount lustre-OST0000 complete [ 1904.097908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.103572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1904.106326] Lustre: Skipped 2 previous similar messages [ 1904.121573] Lustre: Skipped 2 previous similar messages [ 1909.218079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1909.222651] Lustre: Skipped 1 previous similar message [ 1910.136490] Lustre: server umount lustre-MDT0000 complete [ 1913.596345] LustreError: 74379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735581 with bad export cookie 15969255508629131673 [ 1913.597237] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1913.610665] LustreError: 74379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1913.949517] Lustre: server umount lustre-MDT0001 complete [ 1922.263509] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1924.519364] Key type lgssc unregistered [ 1924.795632] LNet: 78184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1924.807811] LNetError: 78184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1924.823950] LNet: Removed LNI 192.168.203.119@tcp [ 1925.465231] Key type .llcrypt unregistered [ 1925.467669] Key type ._llcrypt unregistered [ 1944.309352] Key type ._llcrypt registered [ 1944.312946] Key type .llcrypt registered [ 1944.381768] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1945.167922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1945.201346] alg: No test for adler32 (adler32-zlib) [ 1946.210727] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 1946.449845] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1948.095322] Key type lgssc registered [ 1949.203731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1955.831920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1955.852568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.298383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.064721] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1967.588532] LustreError: 79575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.603165] LustreError: 79575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1968.119514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.578573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1971.769950] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1978.071510] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1982.602591] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1987.869385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1988.294970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.032571] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2000.117452] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2010.162180] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2010.832676] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2012.381160] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2012.386514] Lustre: Skipped 1 previous similar message [ 2014.683611] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2014.694596] Lustre: Skipped 2 previous similar messages [ 2018.686855] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2018.697630] Lustre: Skipped 4 previous similar messages [ 2027.332323] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2027.341028] Lustre: Skipped 9 previous similar messages [ 2043.481731] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2043.486610] Lustre: Skipped 17 previous similar messages [ 2076.038118] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2076.042043] Lustre: Skipped 39 previous similar messages [ 2099.189954] Lustre: 87670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2101.685760] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:16:08 (1787735768) [ 2103.167911] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2104.730246] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:16:11 (1787735771) [ 2110.433349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2110.447548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.453446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2110.461970] Lustre: Skipped 1 previous similar message [ 2110.482314] Lustre: Skipped 1 previous similar message [ 2114.303551] Lustre: server umount lustre-OST0000 complete [ 2120.673175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.683434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.687622] Lustre: Skipped 1 previous similar message [ 2124.152443] Lustre: server umount lustre-MDT0000 complete [ 2125.794994] LustreError: 79575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.817526] LustreError: 79575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2127.683568] LustreError: 82756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735795 with bad export cookie 1062066337672639168 [ 2127.688632] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2127.692654] LustreError: 82756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2127.992333] Lustre: server umount lustre-MDT0001 complete [ 2136.632319] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2139.451632] Key type lgssc unregistered [ 2139.731685] LNet: 88922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.741431] LNetError: 88922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.768655] LNet: Removed LNI 192.168.203.119@tcp [ 2140.653731] Key type .llcrypt unregistered [ 2140.658974] Key type ._llcrypt unregistered [ 2145.126641] Key type ._llcrypt registered [ 2145.129438] Key type .llcrypt registered [ 2145.227598] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2150.388768] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules [ 2151.754467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2151.783775] alg: No test for adler32 (adler32-zlib) [ 2152.989444] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 2153.243138] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2154.999540] Key type lgssc registered [ 2156.594164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2157.981405] LustreError: 90126:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2175.795149] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2179.822459] LustreError: 90126:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2185.652586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2185.675572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.133797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.856882] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2196.453700] LustreError: 91150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2196.486052] LustreError: 91150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2197.664973] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:17:44 (1787735864) [ 2201.491676] LustreError: 91183:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2201.499610] LustreError: 91183:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2201.505682] LustreError: 91183:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 2201.741233] Lustre: server umount lustre-MDT0000 complete [ 2213.063426] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2215.724799] Key type lgssc unregistered [ 2215.972494] LNet: 92678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.980894] LNetError: 92678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.998802] LNet: Removed LNI 192.168.203.119@tcp [ 2216.640838] Key type .llcrypt unregistered [ 2216.645073] Key type ._llcrypt unregistered [ 2234.683704] Key type ._llcrypt registered [ 2234.691141] Key type .llcrypt registered [ 2234.832757] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2236.047665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2236.205419] alg: No test for adler32 (adler32-zlib) [ 2237.329516] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 2237.641299] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2239.359177] Key type lgssc registered [ 2240.432953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2247.925868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2247.944987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.423788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.310378] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2259.940197] LustreError: 93894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.961469] LustreError: 93894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2260.390260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.965222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2264.751235] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2272.836656] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2278.268940] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2285.705610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.147270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.475257] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2300.625332] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2341.516187] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:20:08 (1787736008) [ 2343.048659] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2344.422675] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:20:11 (1787736011) [ 2348.000876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2348.013340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.022543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2350.565149] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.565616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2350.572448] Lustre: Skipped 1 previous similar message [ 2350.583432] Lustre: Skipped 2 previous similar messages [ 2353.911966] Lustre: server umount lustre-MDT0000 complete [ 2355.684812] LustreError: 93889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.704571] LustreError: 93889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2357.870907] LustreError: 93874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787736025 with bad export cookie 7326215894098437593 [ 2357.877317] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2357.893810] LustreError: 93874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2358.256080] Lustre: server umount lustre-MDT0001 complete [ 2371.566584] Lustre: server umount lustre-OST0000 complete [ 2383.052634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.388748] LustreError: 98167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.600338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2387.375758] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2388.962812] LustreError: 98168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.087194] LustreError: 98167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.834731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.332199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2398.890711] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2405.117747] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2410.366720] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2421.212037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2421.379295] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2421.385306] LustreError: 98156:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2421.429048] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2421.440447] LustreError: 100251:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2421.453869] LustreError: 100251:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2421.460204] LustreError: 100251:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2421.539281] Lustre: server umount lustre-OST0000 complete [ 2421.542767] LustreError: 100251:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2429.236175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2429.364677] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2429.370279] LustreError: 98155:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2429.387406] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2429.393548] LustreError: 100716:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2429.397494] LustreError: 100716:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2429.401282] LustreError: 100716:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2429.442727] Lustre: server umount lustre-OST0000 complete [ 2429.444827] LustreError: 100716:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2437.831508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2438.002047] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2438.004602] Lustre: Found index 0 for lustre-OST0000, updating log [ 2438.013226] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2438.092384] Lustre: lustre-OST0000: new disk, initializing [ 2438.099327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2438.112837] Lustre: 101249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2438.436700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.456330] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2446.906126] Lustre: Failing over lustre-OST0000 [ 2447.088689] Lustre: server umount lustre-OST0000 complete [ 2450.920458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.922138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.953060] Lustre: Skipped 1 previous similar message [ 2450.966253] Lustre: Skipped 1 previous similar message [ 2456.033531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2456.039837] Lustre: Skipped 1 previous similar message [ 2456.602540] Lustre: server umount lustre-MDT0000 complete [ 2459.938449] LustreError: 98148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787736127 with bad export cookie 7326215894098440337 [ 2459.944067] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2459.958245] LustreError: 98148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2469.644480] Lustre: server umount lustre-MDT0001 complete [ 2477.900926] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:22:24 (1787736144) [ 2493.572526] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2501.523149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.837626] LustreError: 103609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.929964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.116388] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2514.069797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.566657] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2521.102477] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2526.865350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2527.386050] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2532.833728] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2532.838566] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2532.855069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2533.797720] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2537.952624] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2537.962976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2540.011425] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 2540.022955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2540.055109] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2540.055334] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2541.638913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2541.722611] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2541.785612] Lustre: lustre-OST0001: new disk, initializing [ 2541.788199] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2541.795847] Lustre: 105927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2542.063326] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2542.073112] Lustre: Skipped 2 previous similar messages [ 2542.406287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2542.412927] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2542.491292] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2547.697869] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2554.038452] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2560.656780] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2565.369390] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.19@tcp (stopping) [ 2566.128894] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2566.133663] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.308383] Lustre: server umount lustre-MDT0001 complete [ 2573.310744] LustreError: 103610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.334323] LustreError: 103610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2582.634241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.812705] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2582.824255] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2582.912982] Lustre: 103598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2584.854857] LustreError: 107501:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2584.863799] LustreError: 107501:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2584.877632] LustreError: 103602:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2585.061702] LustreError: 107510:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.119@tcp: cfg command failed: rc = -17 [ 2585.084198] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2585.084198] [ 2588.450294] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2588.679938] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2596.823097] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:24:23 (1787736263) [ 2611.804757] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2626.179565] Lustre: 109539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2648.711992] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2653.203712] Lustre: 110674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2737.121717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2737.125320] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.129198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2737.129207] Lustre: Skipped 5 previous similar messages [ 2737.152084] Lustre: Skipped 5 previous similar messages [ 2739.472996] Lustre: server umount lustre-MDT0000 complete [ 2742.244734] LustreError: 111019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.260853] LustreError: 111019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2743.119393] LustreError: 105926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787736410 with bad export cookie 7326215894098441268 [ 2743.125394] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.128072] LustreError: 105926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2743.534429] Lustre: server umount lustre-MDT0001 complete [ 2756.664289] Lustre: server umount lustre-OST0000 complete [ 2770.364852] Lustre: server umount lustre-OST0001 complete [ 2775.586705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2784.108763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2791.114911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2799.073414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2815.585330] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2823.508520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.541896] Lustre: lustre-MDT0000: reset Object Index mappings [ 2823.996215] LustreError: 114626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.142734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.147771] Lustre: Skipped 1 previous similar message [ 2828.930624] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2836.665394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.687051] Lustre: lustre-MDT0001: reset Object Index mappings [ 2841.621366] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2844.369780] Lustre: 115773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2850.704235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2857.164321] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2862.644687] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 2865.034684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.797458] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2868.800356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 2868.809520] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2871.490378] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2878.998523] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2882.017584] Lustre: 117651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2929.908638] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:29:55 (1787736595) [ 2955.756685] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.772872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2955.777587] Lustre: Skipped 2 previous similar messages [ 2955.801095] Lustre: Skipped 6 previous similar messages [ 2962.264185] Lustre: server umount lustre-MDT0000 complete [ 2965.996821] LustreError: 115699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.020253] LustreError: 115699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2966.201167] LustreError: 114608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787736634 with bad export cookie 7326215894098488161 [ 2966.207732] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2966.223097] LustreError: 114608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2972.791710] Lustre: server umount lustre-MDT0001 complete [ 2982.823121] Lustre: server umount lustre-OST0000 complete [ 3001.311860] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3001.474880] Lustre: server umount lustre-OST0001 complete [ 3008.638755] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 3019.120422] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3056.378519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.616693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3056.657505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3056.776686] Lustre: lustre-MDT0000: new disk, initializing [ 3056.864395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.874748] Lustre: Skipped 3 previous similar messages [ 3056.884718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3062.064121] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3069.963363] Lustre: server umount lustre-MDT0000 complete [ 3079.079467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.375796] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3079.396284] Lustre: 121919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3079.419360] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3084.643043] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3090.003393] Lustre: server umount lustre-MDT0000 complete [ 3098.529302] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:32:45 (1787736765) [ 3099.858962] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3101.637480] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:32:48 (1787736768) [ 3118.573146] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3127.896641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.740132] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3140.523903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.646352] Lustre: 123969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3140.724102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3140.736033] Lustre: Skipped 1 previous similar message [ 3140.883875] Lustre: lustre-MDT0001: new disk, initializing [ 3141.040702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3141.074401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3145.115500] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3149.572508] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3156.232702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3156.604297] Lustre: lustre-OST0000: new disk, initializing [ 3156.609148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3156.615676] Lustre: 125547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3158.662647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3158.675254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3158.810765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3162.740588] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3172.928654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3173.086960] Lustre: lustre-OST0001: new disk, initializing [ 3173.092679] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3173.098489] Lustre: 126419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3175.226290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3175.238042] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3175.334702] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3178.452571] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3187.290159] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3190.830883] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3584.961898] Lustre: 132836:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3584.967045] Lustre: 132836:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3584.972069] Lustre: 132836:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3584.976240] Lustre: 132836:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3584.983625] Lustre: 132836:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3584.988869] Lustre: 132836:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3881.980169] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:45:47 (1787737547) [ 3898.851228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3898.857291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3898.870857] Lustre: Skipped 3 previous similar messages [ 3898.878730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3898.885787] Lustre: Skipped 6 previous similar messages [ 3899.368492] Lustre: server umount lustre-MDT0000 complete [ 3900.386831] LustreError: 127265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3900.411490] LustreError: 127265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3902.986314] LustreError: 123962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787737570 with bad export cookie 7326215894098555774 [ 3902.993974] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3902.995776] LustreError: 123962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3905.506429] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.518346] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3905.522765] Lustre: Skipped 4 previous similar messages [ 3905.535235] Lustre: Skipped 1 previous similar message [ 3909.475563] Lustre: server umount lustre-MDT0001 complete [ 3919.459086] Lustre: server umount lustre-OST0000 complete [ 3937.247210] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3937.418548] Lustre: server umount lustre-OST0001 complete [ 3943.689908] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 3952.574891] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3986.097678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.385554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3986.414742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3986.487288] Lustre: lustre-MDT0000: new disk, initializing [ 3986.577985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.581613] Lustre: Skipped 5 previous similar messages [ 3986.592569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3991.452599] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4002.787996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.964887] Lustre: 135891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4003.017705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4003.021590] Lustre: Skipped 1 previous similar message [ 4003.085584] Lustre: lustre-MDT0001: new disk, initializing [ 4003.208716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4003.229970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4007.878307] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4016.547624] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4021.928742] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4028.473865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4028.690139] Lustre: lustre-OST0000: new disk, initializing [ 4028.693399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4028.699190] Lustre: 137860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4028.766960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4028.772097] Lustre: Skipped 1 previous similar message [ 4030.710082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4030.724044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4030.810783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4033.772989] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4042.562694] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4051.253460] Lustre: lustre-MDD0000: changelog on [ 4053.605560] Lustre: lustre-MDD0001: changelog on [ 6065.043320] Lustre: lustre-MDD0001: changelog off [ 6067.061342] Lustre: 138579:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6067.891410] Lustre: lustre-MDD0000: changelog off [ 6078.953383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6078.961811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.975920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.998392] Lustre: Skipped 1 previous similar message [ 6079.032076] Lustre: Skipped 3 previous similar messages [ 6084.101473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6084.106803] Lustre: Skipped 1 previous similar message [ 6089.189189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6089.197788] Lustre: Skipped 1 previous similar message [ 6093.279457] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6093.615389] Lustre: server umount lustre-OST0000 complete [ 6099.428072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6099.459083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6104.553120] LustreError: 136773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.571769] LustreError: 136773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6104.663386] Lustre: server umount lustre-MDT0000 complete [ 6108.266770] LustreError: 135882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787739776 with bad export cookie 7326215894099377273 [ 6108.272995] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6108.280543] LustreError: 135882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6108.703803] Lustre: server umount lustre-MDT0001 complete [ 6120.424158] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 6123.519535] Key type lgssc unregistered [ 6123.815667] LNet: 147040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6123.837682] LNetError: 147040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6123.851440] LNet: Removed LNI 192.168.203.119@tcp [ 6124.801367] Key type .llcrypt unregistered [ 6124.803521] Key type ._llcrypt unregistered [ 6147.072767] Key type ._llcrypt registered [ 6147.078091] Key type .llcrypt registered [ 6147.187965] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 6162.988755] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 6164.296939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6164.328906] alg: No test for adler32 (adler32-zlib) [ 6165.439096] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 6165.645485] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 6167.335203] Key type lgssc registered [ 6168.414941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6206.415383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6206.460108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6207.746654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6207.796194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6207.910514] Lustre: lustre-MDT0000: new disk, initializing [ 6208.004209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6208.022492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6213.005139] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6223.882646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6223.978560] Lustre: 150241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6224.038177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6224.042872] Lustre: Skipped 1 previous similar message [ 6224.116897] Lustre: lustre-MDT0001: new disk, initializing [ 6224.260511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6224.318845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6224.336908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6228.836771] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6238.015433] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6244.799781] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6252.148774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6252.539942] Lustre: lustre-OST0000: new disk, initializing [ 6252.548382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6252.559627] Lustre: 152210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6252.697400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6260.450781] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6262.365434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6262.389383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6262.522048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6274.250640] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6283.896476] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_module obdecho/obdecho [ 6287.328494] LustreError: 153215:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6287.339456] LustreError: 153215:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6292.080777] LustreError: 153311:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6306.275035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6306.290279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6306.301557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6308.843663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6308.845316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6308.855583] Lustre: Skipped 1 previous similar message [ 6308.880787] Lustre: Skipped 1 previous similar message [ 6311.956544] Lustre: server umount lustre-MDT0000 complete [ 6315.780563] LustreError: 150233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787739983 with bad export cookie 12205499439997133705 [ 6315.789445] LustreError: 150233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6315.789902] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6316.280768] Lustre: server umount lustre-MDT0001 complete [ 6331.010577] Lustre: server umount lustre-OST0000 complete [ 6354.513165] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 6364.273335] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 6411.925580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.387629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6412.416972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6412.489179] Lustre: lustre-MDT0000: new disk, initializing [ 6412.555022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6412.572822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6417.417279] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6427.447338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.538974] Lustre: 157030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6427.582442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6427.587456] Lustre: Skipped 1 previous similar message [ 6427.681292] Lustre: lustre-MDT0001: new disk, initializing [ 6427.777643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6427.820437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6427.830034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6431.029812] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6439.089452] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6444.639471] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6450.906353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6451.151479] Lustre: lustre-OST0000: new disk, initializing [ 6451.156393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6451.165619] Lustre: 158996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6451.233219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6457.368729] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6457.396259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6457.408910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6457.536461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6466.795198] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6471.739936] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6471.973644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6476.982169] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6477.202432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6483.425273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6483.425432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.428515] LustreError: Skipped 1 previous similar message [ 6483.429211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6483.434276] Lustre: Skipped 1 previous similar message [ 6483.440435] Lustre: Skipped 2 previous similar messages [ 6485.408976] Lustre: server umount lustre-OST0000 complete [ 6493.669043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6493.688326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6493.694725] Lustre: Skipped 1 previous similar message [ 6495.769884] Lustre: server umount lustre-MDT0000 complete [ 6498.789111] LustreError: 157041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6498.802279] LustreError: 157041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6500.223320] LustreError: 157021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787740168 with bad export cookie 12205499439997134664 [ 6500.226566] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6500.233663] LustreError: 157021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6500.505560] Lustre: server umount lustre-MDT0001 complete [ 6503.824404] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:29:30 (1787740170) [ 6506.103869] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6508.538930] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:29:34 (1787740174) [ 6516.976634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6517.660400] LustreError: 161410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6517.735526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.620941] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6522.850850] LustreError: 161411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6527.971117] LustreError: 161410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6533.088664] LustreError: 161411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.143381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6540.738318] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6548.650176] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6554.928673] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6562.029399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6562.224472] LustreError: 163231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6562.518556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6562.538040] Lustre: Skipped 1 previous similar message [ 6569.857498] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6578.397652] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6590.945954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6590.948983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.976367] Lustre: Skipped 1 previous similar message [ 6590.995926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6597.320789] Lustre: server umount lustre-MDT0000 complete [ 6599.902692] LustreError: 161406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6601.646454] LustreError: 161391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787740269 with bad export cookie 12205499439997135546 [ 6601.650510] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6601.666341] LustreError: 161391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6605.049872] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.19@tcp (stopping) [ 6605.065918] Lustre: Skipped 4 previous similar messages [ 6605.793123] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.815500] Lustre: Skipped 2 previous similar messages [ 6608.266658] Lustre: server umount lustre-MDT0001 complete [ 6616.090892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6631.478496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.815879] LustreError: 165267:0:(llog.c:1655:llog_backup()) MGC192.168.203.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6631.828021] Lustre: 165267:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6635.800210] LustreError: 165272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6635.838383] LustreError: 165272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6692.319423] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6692.327725] Lustre: 165267:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.119@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6702.048576] LustreError: 165272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6702.094143] LustreError: 165272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 6733.334970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6738.419521] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6739.119492] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6750.149521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6761.954771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6777.845229] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6779.054573] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6787.188885] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6788.821771] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6795.490747] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6797.325340] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6818.788589] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.798467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6818.817150] Lustre: Skipped 1 previous similar message [ 6818.835721] Lustre: Skipped 4 previous similar messages [ 6820.234295] Lustre: server umount lustre-MDT0000 complete [ 6824.533835] LustreError: 161391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787740492 with bad export cookie 12205499439997142035 [ 6824.549710] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6824.969784] Lustre: server umount lustre-MDT0001 complete [ 6839.885508] Lustre: server umount lustre-OST0000 complete [ 6865.745204] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 6883.205340] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 6928.979822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6929.283902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6929.345890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6929.461795] Lustre: lustre-MDT0000: new disk, initializing [ 6929.578933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6929.585867] Lustre: Skipped 1 previous similar message [ 6929.598821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6935.473651] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6946.223825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6946.393905] Lustre: 170409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6946.715328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6951.833481] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6963.008968] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6968.228961] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6974.813753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6975.090463] Lustre: lustre-OST0000: new disk, initializing [ 6975.098135] Lustre: Skipped 1 previous similar message [ 6975.105275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6975.119671] Lustre: Skipped 2 previous similar messages [ 6975.133740] Lustre: 172376:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6980.936721] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6982.237593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6982.244670] Lustre: Skipped 1 previous similar message [ 6982.256690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6982.355541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6991.064866] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6995.861029] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6996.093041] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7001.114608] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7001.303682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7008.227907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7008.252318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7008.280257] Lustre: Skipped 1 previous similar message [ 7008.294936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7008.310772] Lustre: Skipped 1 previous similar message [ 7010.080391] Lustre: server umount lustre-OST0000 complete [ 7018.464888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7018.471844] LustreError: Skipped 1 previous similar message [ 7022.782988] Lustre: server umount lustre-MDT0000 complete [ 7023.587087] LustreError: 172804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7023.645121] LustreError: 172804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 57 previous similar messages [ 7027.203489] LustreError: 173919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787740695 with bad export cookie 12205499439997158828 [ 7027.205889] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7027.207579] LustreError: 173919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7027.571885] Lustre: server umount lustre-MDT0001 complete [ 7040.966176] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:38:25 (1787740705) [ 7048.639141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7054.316568] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7063.108941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7068.443551] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7076.421149] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7082.539035] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7089.738276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7096.757549] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7107.420596] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7121.451634] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:39:47 (1787740787) [ 7129.058212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7129.060759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7129.073529] LustreError: Skipped 1 previous similar message [ 7129.079043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7129.081929] Lustre: Skipped 4 previous similar messages [ 7129.095504] Lustre: Skipped 3 previous similar messages [ 7131.343151] Lustre: server umount lustre-OST0000 complete [ 7142.537307] Lustre: server umount lustre-MDT0000 complete [ 7146.575662] LustreError: 174770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787740814 with bad export cookie 12205499439997159710 [ 7146.601831] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7147.147298] Lustre: server umount lustre-MDT0001 complete [ 7159.208878] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7162.038660] Key type lgssc unregistered [ 7162.399990] LNet: 178524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7162.419958] LNetError: 178524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7162.444872] LNet: Removed LNI 192.168.203.119@tcp [ 7163.154774] Key type .llcrypt unregistered [ 7163.158072] Key type ._llcrypt unregistered [ 7185.696290] Key type ._llcrypt registered [ 7185.699661] Key type .llcrypt registered [ 7185.769408] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7186.770623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7186.799864] alg: No test for adler32 (adler32-zlib) [ 7188.838881] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 7189.298159] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7191.079994] Key type lgssc registered [ 7193.068320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7202.615085] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7202.661672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7218.271363] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7223.391318] LustreError: 179769:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.119@tcp: failed processing log, type 4: rc = -110 [ 7228.511601] LustreError: 179769:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7228.528852] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7228.535203] LustreError: MGC192.168.203.119@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7228.546273] LustreError: 179769:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7228.563568] LustreError: 179769:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7228.576427] LustreError: 179769:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7228.631814] Lustre: server umount lustre-OST0000 complete [ 7228.641718] LustreError: 179769:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7236.171055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7237.514146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7241.380623] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7247.848432] LustreError: 180267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.866089] LustreError: 180267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7248.408802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7248.687210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7252.780360] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7260.114126] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7265.745502] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7271.635750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7271.792867] LustreError: 182083:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7271.800665] Lustre: 182083:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.119@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7271.947475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7277.451867] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7284.471673] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7288.290943] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7288.303633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7288.305416] Lustre: Skipped 1 previous similar message [ 7288.326152] Lustre: Skipped 2 previous similar messages [ 7292.716764] Lustre: server umount lustre-MDT0000 complete [ 7293.410465] LustreError: 180994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7293.454892] LustreError: 180994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7296.927751] LustreError: 180247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787740964 with bad export cookie 10989270040881739567 [ 7296.932820] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7296.952741] LustreError: 180247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7297.430929] Lustre: server umount lustre-MDT0001 complete [ 7313.930398] Lustre: server umount lustre-OST0000 complete [ 7323.453129] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 7336.496252] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7376.443760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7376.683173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7376.714386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7376.788981] Lustre: lustre-MDT0000: new disk, initializing [ 7376.859169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7376.874066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7380.470457] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7390.038641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.138754] Lustre: 185607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7390.188984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7390.192719] Lustre: Skipped 1 previous similar message [ 7390.260970] Lustre: lustre-MDT0001: new disk, initializing [ 7390.328656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7390.365501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7390.379642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7394.724812] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7403.151630] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7407.961224] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7414.562193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7414.832364] Lustre: lustre-OST0000: new disk, initializing [ 7414.836516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7414.851620] Lustre: 187576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7414.966472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7421.592032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7421.622832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7421.798332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7422.683839] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7432.012587] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7436.269110] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7436.428399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7440.561388] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7440.729784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7447.521681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7447.524843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7447.537300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7447.548662] Lustre: Skipped 2 previous similar messages [ 7447.559958] Lustre: Skipped 1 previous similar message [ 7448.841476] Lustre: server umount lustre-OST0000 complete [ 7457.248534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7457.262594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7457.286047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7459.088356] Lustre: server umount lustre-MDT0000 complete [ 7462.427104] LustreError: 185602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787741130 with bad export cookie 10989270040881740295 [ 7462.428807] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7462.438170] LustreError: 185602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7462.797566] Lustre: server umount lustre-MDT0001 complete [ 7471.504177] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:45:37 (1787741137) [ 7489.481924] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7492.677928] Key type lgssc unregistered [ 7492.957864] LNet: 190524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7492.969853] LNetError: 190524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7492.980990] LNet: Removed LNI 192.168.203.119@tcp [ 7493.987202] Key type .llcrypt unregistered [ 7493.989586] Key type ._llcrypt unregistered [ 7512.523312] Key type ._llcrypt registered [ 7512.528642] Key type .llcrypt registered [ 7512.587895] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7514.165952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7514.197065] alg: No test for adler32 (adler32-zlib) [ 7515.479517] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 7515.801756] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7517.479204] Key type lgssc registered [ 7518.939277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7527.721776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7527.739773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7529.252414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7533.625427] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7539.686373] LustreError: 191746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7539.712173] LustreError: 191746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7542.008787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7542.563219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7547.535219] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7556.647429] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7563.859478] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7571.969802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7572.194586] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7572.198454] LustreError: 193611:0:(llog.c:1696:llog_backup()) MGC192.168.203.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7572.207914] Lustre: 193611:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.119@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7572.337164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7579.653514] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7592.596779] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7598.079180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7598.088387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7598.126565] Lustre: Skipped 1 previous similar message [ 7598.144088] Lustre: Skipped 1 previous similar message [ 7601.911051] Lustre: server umount lustre-MDT0000 complete [ 7603.173944] LustreError: 191741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7603.216137] LustreError: 191741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7607.952095] LustreError: 191727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787741275 with bad export cookie 13742840207450224491 [ 7607.953941] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7607.961599] LustreError: 191727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7608.308462] LustreError: 193137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7608.317297] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7608.680358] Lustre: server umount lustre-MDT0001 complete [ 7615.119667] Lustre: server umount lustre-OST0000 complete [ 7628.806514] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 7640.023819] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7677.497051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.736796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7677.764588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7677.839464] Lustre: lustre-MDT0000: new disk, initializing [ 7677.917813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.948127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7683.111784] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7693.861540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7693.993838] Lustre: 197135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7694.042665] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7694.044404] Lustre: Skipped 1 previous similar message [ 7694.047744] LustreError: 197916:0:(llog.c:1696:llog_backup()) MGC192.168.203.119@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7694.070048] LustreError: 197916:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7694.081189] Lustre: 197916:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.119@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7694.121608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7694.129191] Lustre: Skipped 1 previous similar message [ 7694.254494] Lustre: lustre-MDT0001: new disk, initializing [ 7694.373487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7694.428500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7694.450744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7699.903686] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7710.214803] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7716.365990] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7722.506603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7722.708147] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7722.711015] Lustre: Skipped 1 previous similar message [ 7722.714268] LustreError: 199078:0:(llog.c:1696:llog_backup()) MGC192.168.203.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7722.723793] LustreError: 199078:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7722.732216] Lustre: 199078:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.119@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7722.745714] Lustre: 199078:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7722.768716] Lustre: lustre-OST0000: new disk, initializing [ 7722.772322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7722.778545] Lustre: 199101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7722.838078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7728.968455] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7729.250846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7729.262501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7729.437786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7738.614310] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7743.127929] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7743.335753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7748.324312] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7748.566864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7755.236776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.244159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7755.246623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7755.268350] Lustre: Skipped 1 previous similar message [ 7755.286788] Lustre: Skipped 2 previous similar messages [ 7756.907411] Lustre: server umount lustre-OST0000 complete [ 7765.472645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.484261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7767.986207] Lustre: server umount lustre-MDT0000 complete [ 7770.593164] LustreError: 197145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7770.616471] LustreError: 197145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7772.851212] LustreError: 197128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787741440 with bad export cookie 13742840207450225212 [ 7772.866708] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7772.870968] LustreError: 197128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7773.283795] Lustre: server umount lustre-MDT0001 complete [ 7783.884135] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:50:49 (1787741449) [ 7800.119330] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7810.098061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7810.500984] LustreError: 202230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.610332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7815.179867] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7820.771921] LustreError: 202230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.801248] LustreError: 202230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7823.869996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7829.170251] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7832.403210] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7839.795353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7846.974625] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7856.302884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7856.506601] Lustre: lustre-OST0001: new disk, initializing [ 7856.513321] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7856.521172] Lustre: 204540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7856.600296] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7856.604011] Lustre: Skipped 2 previous similar messages [ 7862.421083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7862.433837] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7862.558806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7864.248560] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7876.277712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7883.027438] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7885.512187] Lustre: DEBUG MARKER: ADD OST3 [ 7892.345448] loop: module loaded [ 7892.427248] loop0: detected capacity change from 0 to 204800000 [ 7892.678634] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7892.705579] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7892.720332] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7900.524366] Lustre: DEBUG MARKER: STOP OST3 [ 7900.963625] loop0: detected capacity change from 0 to 204800000 [ 7907.582823] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7907.935490] Lustre: lustre-OST0003: new disk, initializing [ 7907.942745] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7907.953901] Lustre: 206597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7909.462313] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7909.573551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7909.589921] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7909.632214] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7911.233238] Lustre: Failing over lustre-OST0003 [ 7911.520095] LustreError: 206700:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7911.589143] Lustre: server umount lustre-OST0003 complete [ 7914.760732] LustreError: 204496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7914.991576] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7914.992298] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7914.997356] LustreError: Skipped 1 previous similar message [ 7915.022307] Lustre: Skipped 2 previous similar messages [ 7916.213664] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7926.970184] Lustre: DEBUG MARKER: START OST3 again [ 7933.165278] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7933.475471] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7933.485872] Lustre: Skipped 1 previous similar message [ 7933.509197] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7934.828199] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7935.444939] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7935.445942] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7938.948183] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7958.632655] Lustre: Failing over lustre-OST0003 [ 7958.795888] Lustre: server umount lustre-OST0003 complete [ 7959.036653] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.060087] LustreError: 203721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7959.081951] LustreError: 203721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7977.952690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7977.962434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7977.969765] Lustre: Skipped 1 previous similar message [ 7977.975324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7977.978184] Lustre: Skipped 1 previous similar message [ 7986.209353] Lustre: server umount lustre-MDT0000 complete [ 7990.625607] LustreError: 202210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787741658 with bad export cookie 13742840207450226094 [ 7990.658502] LustreError: 202210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7990.662576] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7994.851688] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7994.855037] Lustre: Skipped 5 previous similar messages [ 7997.817544] Lustre: server umount lustre-MDT0001 complete [ 8004.584238] Lustre: server umount lustre-OST0000 complete [ 8011.673695] Lustre: server umount lustre-OST0001 complete [ 8022.081872] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 8031.554142] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8077.058776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8077.255289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8077.279346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8077.344418] Lustre: lustre-MDT0000: new disk, initializing [ 8077.402642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8077.421183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8081.882663] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8092.349946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8092.453950] Lustre: 212158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8092.756387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8097.865558] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8106.975746] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8113.089611] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8120.424568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8120.680673] Lustre: lustre-OST0000: new disk, initializing [ 8120.685072] Lustre: Skipped 1 previous similar message [ 8120.688805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8120.699624] Lustre: Skipped 2 previous similar messages [ 8120.709798] Lustre: 214125:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8122.505520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8122.522462] Lustre: Skipped 1 previous similar message [ 8122.531769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8122.694088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8128.208102] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8137.536292] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8143.489295] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8143.805842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8148.951074] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8149.191893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8153.575557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8153.589807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8153.593176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8153.602922] Lustre: Skipped 6 previous similar messages [ 8153.652399] Lustre: Skipped 3 previous similar messages [ 8157.983869] Lustre: server umount lustre-OST0000 complete [ 8169.399769] Lustre: server umount lustre-MDT0000 complete [ 8173.906768] LustreError: 212149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787741841 with bad export cookie 13742840207450228390 [ 8173.908204] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8173.929954] LustreError: 212149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8174.419418] Lustre: server umount lustre-MDT0001 complete [ 8178.368699] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:57:24 (1787741844) [ 8202.149390] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 8214.162857] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8251.653024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8251.899586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8251.941720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8252.128894] Lustre: lustre-MDT0000: new disk, initializing [ 8252.287885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8258.281545] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8271.996292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8272.138699] Lustre: 218736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8272.646907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8277.314162] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8287.444656] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8293.289456] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8299.720669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8300.017097] Lustre: 220707:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8301.365894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8306.238878] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8315.179909] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8321.099594] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8321.429195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8326.508247] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8326.800209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8332.279854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8332.284246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8332.304590] Lustre: Skipped 2 previous similar messages [ 8332.317852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8332.333327] LustreError: Skipped 2 previous similar messages [ 8332.342380] Lustre: Skipped 5 previous similar messages [ 8335.154246] Lustre: server umount lustre-OST0000 complete [ 8345.998082] Lustre: server umount lustre-MDT0000 complete [ 8347.616345] LustreError: 218748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8347.675480] LustreError: 218748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 8350.604833] LustreError: 218730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742018 with bad export cookie 13742840207450229272 [ 8350.606811] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8350.611636] LustreError: 218730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8350.884771] Lustre: server umount lustre-MDT0001 complete [ 8359.419198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8360.289880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8360.299700] Lustre: Skipped 5 previous similar messages [ 8365.268323] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8377.072391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.640808] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8392.994819] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8400.341677] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8409.376118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8417.025984] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8424.419638] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8440.306852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8440.308321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8440.345277] Lustre: Skipped 3 previous similar messages [ 8445.650072] Lustre: server umount lustre-OST0000 complete [ 8455.812703] Lustre: server umount lustre-MDT0000 complete [ 8460.184058] LustreError: 224781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742127 with bad export cookie 13742840207450230154 [ 8460.187388] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8460.205750] LustreError: 224781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8460.840797] Lustre: server umount lustre-MDT0001 complete [ 8566.167960] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 8569.392353] Key type lgssc unregistered [ 8569.851908] LNet: 226629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8569.854737] LNetError: 226629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8570.959107] LNet: Removed LNI 192.168.203.119@tcp [ 8572.385358] Key type .llcrypt unregistered [ 8572.389430] Key type ._llcrypt unregistered [ 8601.302393] Key type ._llcrypt registered [ 8601.305876] Key type .llcrypt registered [ 8601.445075] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 8617.554840] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8619.501515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8619.633046] alg: No test for adler32 (adler32-zlib) [ 8621.666456] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 8622.172255] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 8623.967965] Key type lgssc registered [ 8625.380685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8670.643808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8670.687242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8672.000216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8672.042666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8672.124215] Lustre: lustre-MDT0000: new disk, initializing [ 8672.225209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8672.245451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8676.681416] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8687.654196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.833156] Lustre: 229989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8687.871979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8687.875389] Lustre: Skipped 1 previous similar message [ 8687.924918] Lustre: lustre-MDT0001: new disk, initializing [ 8687.993583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8688.028314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8688.042558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8692.801218] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8704.511379] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8710.697127] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8718.413689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8718.823800] Lustre: lustre-OST0000: new disk, initializing [ 8718.833624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8718.843977] Lustre: 231957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8719.037661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8725.553046] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8727.079532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8727.086261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8727.209490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8735.385862] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8742.026927] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8742.385146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8748.814934] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8749.091718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8753.124758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8753.125385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8753.130091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8753.151361] Lustre: Skipped 1 previous similar message [ 8753.161066] LustreError: Skipped 1 previous similar message [ 8758.263545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8758.273033] Lustre: Skipped 1 previous similar message [ 8759.004499] Lustre: server umount lustre-OST0000 complete [ 8764.897495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8764.920354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.965377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8764.982164] Lustre: Skipped 1 previous similar message [ 8768.489155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8768.491419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8772.861281] Lustre: server umount lustre-MDT0000 complete [ 8776.709286] LustreError: 229982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742444 with bad export cookie 15566407915467907619 [ 8776.713197] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8776.719263] LustreError: 229982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8777.038685] Lustre: server umount lustre-MDT0001 complete [ 8795.258777] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8805.496517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8805.997893] LustreError: 234894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8806.022556] LustreError: 234894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8806.070387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8811.490098] LustreError: 234895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8811.510680] LustreError: 234895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8811.719237] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8815.587207] LustreError: 234894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8820.710787] LustreError: 234895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8821.650787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8822.308404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8827.991752] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8831.811451] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8839.046977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8839.473829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8846.602983] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8855.331601] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8855.488285] Lustre: lustre-OST0001: new disk, initializing [ 8855.490757] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8855.498099] Lustre: 237202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8855.592713] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8861.525828] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8865.477414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8865.495519] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8865.673392] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8872.861749] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8880.503493] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8894.268205] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:09:20 (1787742560) [ 8896.484308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8896.494910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8896.502056] Lustre: Skipped 2 previous similar messages [ 8896.516079] Lustre: Skipped 4 previous similar messages [ 8902.807057] Lustre: server umount lustre-MDT0000 complete [ 8906.737130] LustreError: 234889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8906.767617] LustreError: 234889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8907.388883] LustreError: 236402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742575 with bad export cookie 15566407915467908501 [ 8907.395547] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8907.416296] LustreError: 236402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8908.077886] Lustre: server umount lustre-MDT0001 complete [ 8923.405614] Lustre: server umount lustre-OST0000 complete [ 8927.199446] Lustre: 228215:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787742579/real 1787742579] req@ffff99d950479880 x1874583658546048/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787742595 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8927.225474] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8927.239083] Lustre: Skipped 1 previous similar message [ 8927.927172] Lustre: server umount lustre-OST0001 complete [ 8935.679054] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 8945.012668] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8987.817843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8988.307535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8988.339072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8988.459561] Lustre: lustre-MDT0000: new disk, initializing [ 8988.570434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8988.597407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8993.749233] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9006.276173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9006.406820] Lustre: 241146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9006.515924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9006.525319] Lustre: Skipped 1 previous similar message [ 9006.607332] Lustre: lustre-MDT0001: new disk, initializing [ 9006.774348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9006.803348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9012.475699] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9022.717475] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9028.622377] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9035.333464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9035.621198] Lustre: lustre-OST0000: new disk, initializing [ 9035.625572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9035.635042] Lustre: 243115:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9036.908067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9036.922326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9037.018289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9042.126650] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9051.786815] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9056.753308] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9057.064494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9062.127641] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9062.407857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9068.001130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9068.008812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9068.024095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9068.025706] Lustre: Skipped 1 previous similar message [ 9068.034954] Lustre: Skipped 5 previous similar messages [ 9070.983409] Lustre: server umount lustre-OST0000 complete [ 9081.082629] Lustre: server umount lustre-MDT0000 complete [ 9081.313020] LustreError: 242232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9081.345090] LustreError: 242232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9085.070296] LustreError: 241949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742752 with bad export cookie 15566407915467910636 [ 9085.079699] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9085.081718] LustreError: 241949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9085.425243] Lustre: server umount lustre-MDT0001 complete [ 9094.193142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.866478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9094.875144] Lustre: Skipped 2 previous similar messages [ 9099.443526] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9100.276580] LustreError: 245331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9100.301381] LustreError: 245331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9107.920324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.918921] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9120.403080] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9126.057715] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9133.034312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9139.374966] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9146.466181] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9293.283387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9293.287046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9293.306041] LustreError: Skipped 2 previous similar messages [ 9293.308118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9293.323039] Lustre: Skipped 3 previous similar messages [ 9293.326225] Lustre: Skipped 2 previous similar messages [ 9306.085201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9306.306905] Lustre: server umount lustre-OST0000 complete [ 9313.253927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9317.381781] Lustre: server umount lustre-MDT0000 complete [ 9318.885468] LustreError: 246348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9318.913714] LustreError: 246348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9321.113925] LustreError: 245312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742988 with bad export cookie 15566407915467911518 [ 9321.120765] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9321.130688] LustreError: 245312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9321.617691] Lustre: server umount lustre-MDT0001 complete [ 9415.955417] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 9419.387289] Key type lgssc unregistered [ 9419.706925] LNet: 248947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9419.721440] LNetError: 248947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9419.744633] LNet: Removed LNI 192.168.203.119@tcp [ 9420.722254] Key type .llcrypt unregistered [ 9420.724884] Key type ._llcrypt unregistered [ 9445.918851] Key type ._llcrypt registered [ 9445.921639] Key type .llcrypt registered [ 9446.029796] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 9461.773343] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9463.199280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9463.316213] alg: No test for adler32 (adler32-zlib) [ 9464.645778] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 9464.975506] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 9466.719317] Key type lgssc registered [ 9468.117833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9518.745567] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9530.649548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9530.720510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9532.008739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9532.033906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9532.180983] Lustre: lustre-MDT0000: new disk, initializing [ 9532.317885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9532.354423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9537.592696] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9548.441407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9548.622377] Lustre: 253019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9548.681914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9548.688951] Lustre: Skipped 1 previous similar message [ 9548.765946] Lustre: lustre-MDT0001: new disk, initializing [ 9548.907029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9548.948787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9548.963498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9553.911076] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9559.254757] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9566.546879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9566.922319] Lustre: lustre-OST0000: new disk, initializing [ 9566.931881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9566.946240] Lustre: 254651:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9567.220753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9575.565503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9575.581075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9575.760300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9576.374447] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9590.818557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9591.114422] Lustre: lustre-OST0001: new disk, initializing [ 9591.118889] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9591.131759] Lustre: 255522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9591.288369] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9599.205393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9599.223115] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9599.437094] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9602.204435] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9617.062696] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9625.850836] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9640.420696] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9640.428147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9640.429223] Lustre: Skipped 2 previous similar messages [ 9640.443371] Lustre: Skipped 3 previous similar messages [ 9640.492146] Lustre: server umount lustre-MDT0000 complete [ 9644.280977] LustreError: 253013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787743312 with bad export cookie 13515324106667276250 [ 9644.287898] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9644.294620] LustreError: 253013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9644.704791] Lustre: server umount lustre-MDT0001 complete [ 9658.710338] Lustre: server umount lustre-OST0000 complete [ 9661.414510] Lustre: 250526:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787743313/real 1787743313] req@ffff99d84be8d180 x1874584542462592/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787743329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9661.476755] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9661.552878] Lustre: Skipped 1 previous similar message [ 9664.060759] Lustre: server umount lustre-OST0001 complete [ 9685.771595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9699.580817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9710.666912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9726.194748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9749.408934] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9759.880027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9759.905768] Lustre: lustre-MDT0000: reset Object Index mappings [ 9760.393321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9765.014215] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9765.872493] LustreError: 260539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9765.882540] LustreError: 260539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9770.978623] LustreError: 260540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9774.715837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9774.750546] Lustre: lustre-MDT0001: reset Object Index mappings [ 9775.244716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9783.242211] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9788.068761] Lustre: 261680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9797.563769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9797.929575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9799.991607] LustreError: 262035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9805.292930] LustreError: 262157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9805.405573] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9806.320302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9811.452163] LustreError: 262036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9811.471765] LustreError: 262036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9815.351734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9821.197278] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9822.090636] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9830.850989] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9835.112779] Lustre: 263552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9841.281881] Lustre: server umount lustre-MDT0000 complete [ 9841.668813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.676519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9841.685416] LustreError: 262392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9841.696739] Lustre: Skipped 1 previous similar message [ 9845.845864] LustreError: 260519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787743513 with bad export cookie 13515324106667283047 [ 9845.854289] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9845.863685] LustreError: 260519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9846.357143] Lustre: server umount lustre-MDT0001 complete [ 9862.559185] Lustre: 250525:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787743514/real 1787743514] req@ffff99d8442e0a80 x1874584542514560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787743530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9862.596619] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9862.610845] Lustre: Skipped 2 previous similar messages [ 9862.681773] Lustre: server umount lustre-OST0000 complete [ 9866.907171] Lustre: server umount lustre-OST0001 complete [ 9889.350125] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:25:55 (1787743555) [ 9905.536117] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 9916.120415] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9952.315771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9952.559942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9952.577126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9952.664798] Lustre: lustre-MDT0000: new disk, initializing [ 9952.737276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9952.742190] Lustre: Skipped 1 previous similar message [ 9952.754966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9956.968470] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9967.386401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9967.551533] Lustre: 267805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9967.651107] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9967.658103] Lustre: Skipped 1 previous similar message [ 9967.773284] Lustre: lustre-MDT0001: new disk, initializing [ 9967.924559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9967.947156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9972.189368] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9980.709868] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9986.395667] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9993.054950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9993.399829] Lustre: lustre-OST0000: new disk, initializing [ 9993.402565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9993.412647] Lustre: 269775:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9994.681145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9994.687963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9994.796933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9999.594214] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10009.046782] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10013.952661] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10014.119249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10018.683956] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10018.871623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10025.440953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10025.442066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10025.451379] LustreError: Skipped 1 previous similar message [10025.464374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10030.570709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10030.573406] Lustre: Skipped 1 previous similar message [10035.167227] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10035.381311] Lustre: server umount lustre-OST0000 complete [10039.777689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10039.788574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10039.798234] Lustre: Skipped 1 previous similar message [10039.801430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10039.804211] Lustre: Skipped 1 previous similar message [10045.920330] LustreError: 268971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10045.941943] LustreError: 268971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10047.310556] Lustre: server umount lustre-MDT0000 complete [10050.879842] LustreError: 267797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787743718 with bad export cookie 13515324106667285140 [10050.885762] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10050.901625] LustreError: 267797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10051.274498] Lustre: server umount lustre-MDT0001 complete [10069.266230] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10080.875106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10081.595376] LustreError: 272710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10081.666708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10081.669271] Lustre: Skipped 2 previous similar messages [10087.543230] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10097.968329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10103.250514] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10106.154900] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10113.488590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10120.423676] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10129.753374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10129.994476] Lustre: lustre-OST0001: new disk, initializing [10130.001358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10130.008710] Lustre: 275018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10131.594833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10131.601846] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10131.734424] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10136.641348] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10146.260598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10149.945773] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10156.227143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10156.458584] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10165.223898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10165.236251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10165.243105] Lustre: Skipped 1 previous similar message [10165.247040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10165.255011] Lustre: Skipped 2 previous similar messages [10169.981681] Lustre: server umount lustre-MDT0000 complete [10172.905813] LustreError: 273438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10172.925256] LustreError: 273438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10174.150453] LustreError: 272690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787743841 with bad export cookie 13515324106667286022 [10174.161536] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10174.179014] LustreError: 272690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10174.753951] Lustre: server umount lustre-MDT0001 complete [10190.062273] Lustre: server umount lustre-OST0000 complete [10193.375784] Lustre: 250525:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787743845/real 1787743845] req@ffff99d841abb100 x1874584542619904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787743861 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10194.651698] Lustre: server umount lustre-OST0001 complete [10214.428564] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10226.947243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10227.570357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10227.580189] Lustre: Skipped 3 previous similar messages [10232.142627] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10241.718603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10246.555855] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10249.306333] Lustre: 279211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10257.227573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10264.899602] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10273.286728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10275.124928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10280.468251] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10282.581993] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10289.750696] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10293.166278] Lustre: 281088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10297.714855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10297.868608] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10307.987035] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:32:54 (1787743974) [10326.598762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10326.608078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10326.631481] Lustre: Skipped 7 previous similar messages [10326.661123] Lustre: Skipped 7 previous similar messages [10331.006811] Lustre: server umount lustre-MDT0000 complete [10331.618338] LustreError: 278066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10331.673981] LustreError: 278066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10336.691774] LustreError: 278052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787744004 with bad export cookie 13515324106667292350 [10336.725248] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10343.548492] Lustre: server umount lustre-MDT0001 complete [10362.335182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10362.507602] Lustre: server umount lustre-OST0000 complete [10366.373730] LustreError: 250522:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff99d9444a8000 x1874584542700928/t0(0) o38->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10372.610946] Lustre: server umount lustre-OST0001 complete [10379.935823] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [10391.344331] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10444.749837] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10457.632285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10457.817525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10457.837620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10457.896648] Lustre: lustre-MDT0000: new disk, initializing [10457.978652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10463.381495] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10474.527486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10474.728524] Lustre: 285354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10474.936814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10481.033529] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10495.737675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10496.139267] Lustre: 286986:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10496.306984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10496.319796] Lustre: Skipped 5 previous similar messages [10497.546759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10503.524792] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10514.557105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10514.915949] Lustre: 287859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10516.111843] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10516.124172] Lustre: Skipped 1 previous similar message [10516.173680] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10522.866142] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10533.317118] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10537.666101] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10537.674863] Lustre: Skipped 1 previous similar message [10555.222794] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:37:01 (1787744221) [10557.409600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10557.416032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10557.418724] Lustre: Skipped 3 previous similar messages [10557.441831] Lustre: Skipped 7 previous similar messages [10559.674338] Lustre: server umount lustre-MDT0000 complete [10563.667678] LustreError: 285345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787744231 with bad export cookie 13515324106667300204 [10563.676517] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10563.686215] LustreError: 285345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10564.177901] Lustre: server umount lustre-MDT0001 complete [10578.892538] Lustre: server umount lustre-OST0000 complete [10594.564299] Lustre: server umount lustre-OST0001 complete [10603.671611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10603.993787] LustreError: 290103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10604.007490] LustreError: 290103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10608.832158] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10617.183330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10622.425970] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10631.681824] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10641.188188] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10650.603545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10658.178048] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10667.029264] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10704.522548] loop1: detected capacity change from 0 to 204800000 [10704.782125] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10704.797555] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10704.838264] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10711.374165] loop1: detected capacity change from 0 to 204800000 [10717.044735] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10717.136763] LustreError: 290092:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs2781: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10717.163349] LustreError: 290092:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs2781 [10717.169831] LustreError: 290092:0:(mgs_handler.c:517:mgs_target_reg()) fs2781-MDT0001: Error checking target index: rc = -13 [10717.179755] LustreError: fs2781-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10717.192762] LustreError: 293846:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10717.205795] LustreError: 293846:0:(tgt_mount.c:2132:server_put_super()) no obd fs2781-MDT0001 [10717.214730] LustreError: 293846:0:(tgt_mount.c:116:server_deregister_mount()) fs2781-MDT0001 not registered [10717.241071] Lustre: server umount fs2781-MDT0001 complete [10717.243473] LustreError: 293846:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10726.879468] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10727.104985] Lustre: Setting parameter fs2781-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs2781-MDT0001 [10727.184484] Lustre: srv-fs2781-MDT0001: No data found on store. Initialize space. [10727.195435] Lustre: Skipped 4 previous similar messages [10727.307309] Lustre: fs2781-MDT0001: new disk, initializing [10727.314988] Lustre: Skipped 3 previous similar messages [10727.424938] Lustre: srv-fs2781-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10727.435820] Lustre: 294314:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2781-MDT0001-mdtlov: can't create new llog: rc = -115 [10727.452030] LustreError: 294314:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2781-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10732.712453] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10739.877059] Lustre: server umount fs2781-MDT0001 complete [10740.217753] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10750.450426] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10758.466449] loop1: detected capacity change from 0 to 204800000 [10767.624900] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10768.111498] Lustre: 295884:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2781-MDT0001-mdtlov: can't create new llog: rc = -115 [10768.130829] LustreError: 295884:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2781-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10773.006901] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10778.015326] Lustre: server umount fs2781-MDT0001 complete [10791.176959] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [10791.392830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10791.408161] Lustre: Skipped 4 previous similar messages [10803.167184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10803.440120] Lustre: server umount lustre-MDT0000 complete [10818.541738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.783804] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10824.713668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10825.728272] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10834.376372] loop1: detected capacity change from 0 to 204800000 [10839.682806] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10840.009299] Lustre: 298330:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2781-MDT0001-mdtlov: can't create new llog: rc = -115 [10840.021154] LustreError: 298330:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2781-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10844.458510] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10851.083795] Lustre: server umount fs2781-MDT0001 complete [10875.887757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10875.898078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10875.909095] Lustre: Skipped 12 previous similar messages [10879.929046] Lustre: server umount lustre-OST0000 complete [10891.097501] Lustre: server umount lustre-MDT0000 complete [10895.021158] LustreError: 290083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787744562 with bad export cookie 13515324106667302885 [10895.025199] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10895.040982] LustreError: 290083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10895.758635] Lustre: server umount lustre-MDT0001 complete [10909.357129] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [10913.308323] Key type lgssc unregistered [10913.669124] LNet: 300679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10913.680282] LNetError: 300679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10913.712046] LNet: Removed LNI 192.168.203.119@tcp [10914.856577] Key type .llcrypt unregistered [10914.859602] Key type ._llcrypt unregistered [10936.492722] Key type ._llcrypt registered [10936.494965] Key type .llcrypt registered [10936.582304] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10938.075972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10938.249146] alg: No test for adler32 (adler32-zlib) [10939.502876] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [10939.847836] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [10941.558688] Key type lgssc registered [10943.509496] Lustre: Echo OBD driver; http://www.lustre.org/ [10953.284847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10953.302971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10954.907705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10959.755877] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10964.451283] LustreError: 301953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10964.467397] LustreError: 301953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10969.577568] LustreError: 301952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10970.362527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10970.980280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10975.985886] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10986.210340] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10993.857589] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11002.308084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11002.901317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11009.145253] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11016.169740] LustreError: 303782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.838913] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11021.287109] LustreError: 303785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11029.492962] LustreError: 304407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11029.516608] LustreError: 304407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11039.720813] LustreError: 303785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.747063] LustreError: 303785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11043.003923] loop1: detected capacity change from 0 to 204800000 [11043.312140] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11043.323376] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11043.333121] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11049.368671] loop1: detected capacity change from 0 to 204800000 [11055.641964] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11055.815361] Lustre: fs27643-OST0000: new disk, initializing [11055.821930] Lustre: srv-fs27643-OST0000: No data found on store. Initialize space. [11055.828138] Lustre: 305714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11055.835515] LustreError: 303785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11055.846551] LustreError: 303785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11055.943368] Lustre: fs27643-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11063.248784] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11074.139387] Lustre: server umount fs27643-OST0000 complete [11087.588322] loop1: detected capacity change from 0 to 204800000 [11090.922090] LustreError: 303785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11090.943059] LustreError: 303785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11094.932871] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11095.541248] Lustre: fs27643-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11096.044739] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11101.162026] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11104.988712] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11117.618334] loop2: detected capacity change from 0 to 204800000 [11117.752178] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11117.759523] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11117.766361] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11123.744268] loop2: detected capacity change from 0 to 204800000 [11131.487276] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11131.589338] LustreError: fs27643-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11131.610521] LustreError: fs27643-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11131.626733] LustreError: 308757:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11131.637333] LustreError: 308757:0:(tgt_mount.c:2132:server_put_super()) no obd fs27643-OST0001 [11131.648246] LustreError: 308757:0:(tgt_mount.c:116:server_deregister_mount()) fs27643-OST0001 not registered [11131.693400] Lustre: server umount fs27643-OST0001 complete [11131.698406] LustreError: 308757:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11149.309215] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11149.772299] Lustre: fs27643-OST0001: new disk, initializing [11149.776110] Lustre: srv-fs27643-OST0001: No data found on store. Initialize space. [11149.792255] Lustre: 309309:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11149.891579] Lustre: fs27643-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11157.485726] LustreError: 303784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11157.520123] LustreError: 303784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11158.645672] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11170.418375] Lustre: server umount fs27643-OST0000 complete [11185.547833] Lustre: server umount fs27643-OST0001 complete [11201.784638] loop1: detected capacity change from 0 to 204800000 [11209.823977] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11210.209828] Lustre: fs27643-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11219.469351] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11230.097877] loop2: detected capacity change from 0 to 204800000 [11238.059352] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11245.633704] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11249.649854] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11254.770128] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11255.819597] Lustre: server umount fs27643-OST0000 complete [11266.286242] Lustre: server umount fs27643-OST0001 complete [11280.353288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11280.356842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11280.383267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11282.403369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11282.434516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11286.170433] Lustre: server umount lustre-OST0000 complete [11293.665551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11293.671876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11293.678950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11293.692625] Lustre: Skipped 3 previous similar messages [11293.695356] LustreError: Skipped 1 previous similar message [11298.489411] Lustre: server umount lustre-MDT0000 complete [11298.791563] LustreError: 302686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11298.812919] LustreError: 302686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 75 previous similar messages [11303.343762] LustreError: 303807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787744971 with bad export cookie 459550076719258539 [11303.348761] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11303.364107] LustreError: 303807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11303.919768] Lustre: server umount lustre-MDT0001 complete [11318.816951] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11322.169328] Key type lgssc unregistered [11322.543548] LNet: 315239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11322.552510] LNetError: 315239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11322.580160] LNet: Removed LNI 192.168.203.119@tcp [11323.697263] Key type .llcrypt unregistered [11323.702286] Key type ._llcrypt unregistered [11365.207971] Key type ._llcrypt registered [11365.211715] Key type .llcrypt registered [11365.366559] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [11381.657042] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [11382.784568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11382.830627] alg: No test for adler32 (adler32-zlib) [11384.037600] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [11384.287463] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [11386.039427] Key type lgssc registered [11387.066754] Lustre: Echo OBD driver; http://www.lustre.org/ [11426.547512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11426.569695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11427.790598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11427.825673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11427.927380] Lustre: lustre-MDT0000: new disk, initializing [11428.092159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11428.117975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11433.044911] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11445.723333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11445.914669] Lustre: 319101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11446.007316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11446.011450] Lustre: Skipped 1 previous similar message [11446.190680] Lustre: lustre-MDT0001: new disk, initializing [11446.336506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11446.380325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11446.396717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11452.209460] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11464.975889] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11472.995538] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11481.073121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11481.333498] Lustre: lustre-OST0000: new disk, initializing [11481.339626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11481.356461] Lustre: 321076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11481.483532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11487.846540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11487.868210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11488.020472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11490.105713] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11500.966804] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11503.397552] Lustre: Mounted lustre-client - version 2.17.57_80_g6de652b [11504.416640] Lustre: Unmounted lustre-client [11513.826742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11513.840908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11513.854444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11513.873391] Lustre: Skipped 1 previous similar message [11518.946714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11518.958865] Lustre: Skipped 1 previous similar message [11524.078613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11524.091545] Lustre: Skipped 1 previous similar message [11528.159249] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11528.622372] Lustre: server umount lustre-OST0000 complete [11533.280369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11533.291401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11533.308742] Lustre: Skipped 1 previous similar message [11533.315155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11534.308864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.523035] Lustre: server umount lustre-MDT0000 complete [11544.132203] LustreError: 319096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787745211 with bad export cookie 3690930759515307175 [11544.139860] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11544.166124] LustreError: 319096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11544.716883] Lustre: server umount lustre-MDT0001 complete [11554.337369] LNet: 1 local NIs in recovery (showing 1): 192.168.203.119@tcp [11557.135263] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11559.896196] Key type lgssc unregistered [11560.381132] LNet: 323241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11560.389738] LNetError: 323241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11634.165395] LNet: Removed LNI 192.168.203.119@tcp [11635.303077] Key type .llcrypt unregistered [11635.306136] Key type ._llcrypt unregistered [11664.732257] Key type ._llcrypt registered [11664.734255] Key type .llcrypt registered [11664.965517] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [11684.297175] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [11685.855512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11685.904286] alg: No test for adler32 (adler32-zlib) [11687.510349] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [11687.985686] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [11689.847230] Key type lgssc registered [11691.122476] Lustre: Echo OBD driver; http://www.lustre.org/ [11728.679797] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:56:34 (1787745394) [11737.589766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11737.667801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11739.159338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11739.207206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11739.345773] Lustre: lustre-MDT0000: new disk, initializing [11739.526091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11739.561586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11745.390281] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11759.639526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11759.838506] Lustre: 326465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11759.922210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11759.928273] Lustre: Skipped 1 previous similar message [11759.987670] Lustre: lustre-MDT0001: new disk, initializing [11760.101659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11760.162568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11760.178656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11767.354858] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11779.791944] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11786.315408] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11794.252664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11794.647511] Lustre: lustre-OST0000: new disk, initializing [11794.651594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11794.658849] Lustre: 328436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11794.744734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11802.198410] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11802.708100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11802.723690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11802.868258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11812.736251] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11839.028392] Lustre: server umount lustre-OST0000 complete [11847.142949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11847.159372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11847.218107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11849.185088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11849.186112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11849.214951] Lustre: Skipped 1 previous similar message [11854.303743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11854.306663] Lustre: Skipped 1 previous similar message [11854.545995] Lustre: server umount lustre-MDT0000 complete [11855.327094] Lustre: 324639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787745506/real 1787745506] req@00000000f25de3f3 x1874586873405696/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787745522 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11855.378533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.415379] LustreError: 326456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787745527 with bad export cookie 14988788726222283337 [11859.429934] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11859.432136] Lustre: 324640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787745511/real 1787745511] req@0000000042e45a75 x1874586873407360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787745527 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 projid:4294967295 [11859.914213] Lustre: server umount lustre-MDT0001 complete [11874.125239] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11878.242109] Key type lgssc unregistered [11878.679882] LNet: 330347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11878.689636] LNetError: 330347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11879.719159] LNet: Removed LNI 192.168.203.119@tcp [11880.825524] Key type .llcrypt unregistered [11880.829706] Key type ._llcrypt unregistered [11905.927372] Key type ._llcrypt registered [11905.930374] Key type .llcrypt registered [11906.079703] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [11907.014509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11907.176841] alg: No test for adler32 (adler32-zlib) [11908.259543] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [11908.525741] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [11910.215171] Key type lgssc registered [11911.441667] Lustre: Echo OBD driver; http://www.lustre.org/ [11919.821207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11919.849258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11921.425137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11926.169344] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11931.629297] LustreError: 331619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.658665] LustreError: 331619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11935.726619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11936.256559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11940.917634] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11949.763521] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11957.049781] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11965.276942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11965.602826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11973.017846] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11984.316807] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11999.960372] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12002.799820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12007.910106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12007.934325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.950096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12012.001456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12012.002949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12012.013881] Lustre: Skipped 3 previous similar messages [12013.277306] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [12017.124320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12017.138833] Lustre: Skipped 2 previous similar messages [12022.240436] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12022.246669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12022.261388] Lustre: Skipped 3 previous similar messages [12022.469350] Lustre: server umount lustre-MDT0000 complete [12023.512461] LustreError: 331614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12026.531515] LustreError: 331601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787745694 with bad export cookie 7946448921445895403 [12026.533439] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12026.552170] LustreError: 331601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12027.361921] LustreError: 331616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12027.365266] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12027.413995] Lustre: Skipped 1 previous similar message [12031.392604] LustreError: 331615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12031.395225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12031.426746] LustreError: 331615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12031.444255] Lustre: Skipped 2 previous similar messages [12033.027222] Lustre: server umount lustre-MDT0001 complete [12038.421661] Lustre: server umount lustre-OST0000 complete [12049.346757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12054.933491] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12058.307200] Lustre: 335817:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.119@tcp [12061.543439] Lustre: server umount MGS complete [12074.429351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12074.881098] LustreError: 336391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12074.991994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12081.904289] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12084.195344] LustreError: 336392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12084.206936] LustreError: 336392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12094.033258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12094.342820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12100.290873] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12111.226120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12111.726129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12119.117850] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12132.563881] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12134.610407] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12142.773198] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12144.503432] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12150.766603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12150.774279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12150.778329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12150.795101] Lustre: Skipped 1 previous similar message [12150.841477] LustreError: Skipped 1 previous similar message [12151.134373] Lustre: server umount lustre-OST0000 complete [12156.621537] Lustre: server umount lustre-MDT0000 complete [12160.808874] LustreError: 336374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787745828 with bad export cookie 7946448921445896236 [12160.830830] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12163.253328] Lustre: server umount lustre-MDT0001 complete [12177.236674] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12181.566833] Key type lgssc unregistered [12181.965166] LNet: 339833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12182.001591] LNetError: 339833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12182.040500] LNet: Removed LNI 192.168.203.119@tcp [12183.471623] Key type .llcrypt unregistered [12183.488466] Key type ._llcrypt unregistered [12208.074739] Key type ._llcrypt registered [12208.077328] Key type .llcrypt registered [12208.223983] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [12228.515781] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12229.656252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12229.716714] alg: No test for adler32 (adler32-zlib) [12231.352044] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [12231.780973] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [12233.544651] Key type lgssc registered [12235.182967] Lustre: Echo OBD driver; http://www.lustre.org/ [12283.958773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12283.986468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12285.453540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12285.508410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12285.645858] Lustre: lustre-MDT0000: new disk, initializing [12285.724721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12285.741870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12291.174246] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12304.893660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12305.066897] Lustre: 342998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12305.159746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12305.174646] Lustre: Skipped 1 previous similar message [12305.334289] Lustre: lustre-MDT0001: new disk, initializing [12305.545150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12305.660291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12305.677827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12312.016026] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12323.696402] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12330.498850] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12337.905094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12338.223330] Lustre: lustre-OST0000: new disk, initializing [12338.226546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12338.237828] Lustre: 344967:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12338.347607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12344.919878] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12345.956074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12345.974676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12346.215604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12355.846642] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12361.579569] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12361.878957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12367.626346] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12367.958549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12371.936399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12371.942338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12371.956914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12371.978989] Lustre: Skipped 1 previous similar message [12372.004746] Lustre: Skipped 1 previous similar message [12376.383989] Lustre: server umount lustre-OST0000 complete [12382.176539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12382.179220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12382.201461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12382.209139] Lustre: Skipped 1 previous similar message [12387.300478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12387.311779] Lustre: Skipped 2 previous similar messages [12388.013287] Lustre: server umount lustre-MDT0000 complete [12393.082311] LustreError: 344966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787746060 with bad export cookie 8109342865372095142 [12393.086062] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12393.107973] LustreError: 344966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12393.774848] Lustre: server umount lustre-MDT0001 complete [12404.945851] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 08:07:51 (1787746071) [12422.904368] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [12435.255574] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12477.945386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12478.134224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12478.200973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12478.337659] Lustre: lustre-MDT0000: new disk, initializing [12478.475626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12478.519832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12483.969091] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12495.338044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12495.525045] Lustre: 349730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12495.586876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12495.591484] Lustre: Skipped 1 previous similar message [12495.657488] Lustre: lustre-MDT0001: new disk, initializing [12495.770113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12495.834818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12495.851742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12501.383912] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12512.912989] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12521.240343] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12529.738208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12530.120327] Lustre: lustre-OST0000: new disk, initializing [12530.124968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12530.145327] Lustre: 351700:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12530.263796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12539.340613] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12539.580119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12539.612168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12539.886839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12552.549979] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12596.193075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12596.219177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12596.226967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12596.228091] Lustre: Skipped 1 previous similar message [12599.660222] Lustre: server umount lustre-OST0000 complete [12606.438142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12606.441753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12606.458835] Lustre: Skipped 1 previous similar message [12606.468935] Lustre: Skipped 1 previous similar message [12610.712681] Lustre: server umount lustre-MDT0000 complete [12611.554601] LustreError: 349740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12611.595168] LustreError: 349740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12615.561400] LustreError: 349720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787746283 with bad export cookie 8109342865372096017 [12615.584934] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12615.586728] LustreError: 349720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12616.385625] Lustre: server umount lustre-MDT0001 complete [12629.380723] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12632.779908] Key type lgssc unregistered [12633.103994] LNet: 353877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12633.118890] LNetError: 353877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12633.158678] LNet: Removed LNI 192.168.203.119@tcp [12634.123208] Key type .llcrypt unregistered [12634.126229] Key type ._llcrypt unregistered [12743.029949] Key type ._llcrypt registered [12743.038522] Key type .llcrypt registered [12743.294277] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12743.889897] Key type .llcrypt unregistered [12743.906841] Key type ._llcrypt unregistered [12836.148605] Key type ._llcrypt registered [12836.150949] Key type .llcrypt registered [12836.261063] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12836.621243] Key type .llcrypt unregistered [12836.626361] Key type ._llcrypt unregistered [12869.276084] Key type ._llcrypt registered [12869.278362] Key type .llcrypt registered [12869.407790] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12869.671264] Key type .llcrypt unregistered [12869.675489] Key type ._llcrypt unregistered [12908.790304] Key type ._llcrypt registered [12908.793987] Key type .llcrypt registered [12909.082812] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12909.687495] Key type .llcrypt unregistered [12909.690360] Key type ._llcrypt unregistered [12997.586297] Key type ._llcrypt registered [12997.594868] Key type .llcrypt registered [12997.726437] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12998.084499] Key type .llcrypt unregistered [12998.087067] Key type ._llcrypt unregistered [13041.422184] Key type ._llcrypt registered [13041.425559] Key type .llcrypt registered [13041.681385] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13042.381730] Key type .llcrypt unregistered [13042.390942] Key type ._llcrypt unregistered [13075.028311] Key type ._llcrypt registered [13075.030036] Key type .llcrypt registered [13075.156925] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13075.570185] Key type .llcrypt unregistered [13075.575449] Key type ._llcrypt unregistered [13110.689191] Key type ._llcrypt registered [13110.697383] Key type .llcrypt registered [13110.901503] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13111.595601] Key type .llcrypt unregistered [13111.598560] Key type ._llcrypt unregistered [13148.110385] Key type ._llcrypt registered [13148.113320] Key type .llcrypt registered [13148.283309] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13148.713528] Key type .llcrypt unregistered [13148.715623] Key type ._llcrypt unregistered [13185.825479] Key type ._llcrypt registered [13185.828597] Key type .llcrypt registered [13185.976355] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13186.472320] Key type .llcrypt unregistered [13186.475670] Key type ._llcrypt unregistered [13261.169607] Key type ._llcrypt registered [13261.179025] Key type .llcrypt registered [13261.324689] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13261.845429] Key type .llcrypt unregistered [13261.852227] Key type ._llcrypt unregistered [13295.639682] Key type ._llcrypt registered [13295.641322] Key type .llcrypt registered [13295.802619] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13296.097329] Key type .llcrypt unregistered [13296.099344] Key type ._llcrypt unregistered [13341.373478] Key type ._llcrypt registered [13341.375968] Key type .llcrypt registered [13341.523389] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13341.942679] Key type .llcrypt unregistered [13341.945956] Key type ._llcrypt unregistered [13409.078690] Key type ._llcrypt registered [13409.085988] Key type .llcrypt registered [13409.298556] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [13430.481660] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [13432.619896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13432.653368] alg: No test for adler32 (adler32-zlib) [13433.816813] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [13434.160158] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [13435.911376] Key type lgssc registered [13437.559902] Lustre: Echo OBD driver; http://www.lustre.org/ [13480.538634] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13480.593570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13482.070900] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13482.115575] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13482.174354] Lustre: tst250fs-MDT0000: new disk, initializing [13482.287875] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13482.344433] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13487.413724] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [13501.452922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13501.651558] Lustre: 374311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13501.752568] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13501.756842] Lustre: Skipped 1 previous similar message [13501.862898] Lustre: tst250fs-MDT0001: new disk, initializing [13502.032340] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13502.106807] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13502.131118] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13509.510705] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [13516.823244] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13527.037214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13527.370318] Lustre: tst250fs-OST0000: new disk, initializing [13527.376988] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13527.388743] Lustre: 375945:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13527.577762] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13528.888240] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13528.904178] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13529.064761] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13535.061912] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [13549.853046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13550.037028] Lustre: tst250fs-OST0001: new disk, initializing [13550.048296] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13550.062147] Lustre: 376817:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13550.188583] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13557.938737] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13557.953908] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13558.044689] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13558.975350] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [13566.623552] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13569.004734] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13569.016725] Lustre: Skipped 1 previous similar message [13594.081116] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13594.083418] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13594.088045] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13594.110203] Lustre: Skipped 3 previous similar messages [13598.887422] Lustre: server umount tst250fs-MDT0000 complete [13603.982449] LustreError: 374302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787747271 with bad export cookie 4936534230589911523 [13603.989662] LustreError: 374302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13603.994420] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13604.322254] LustreError: 374319:0:(ldlm_lib.c:1190:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13604.325251] Lustre: tst250fs-MDT0001-lwp-OST0000: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13604.335020] LustreError: 374319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [13604.369302] Lustre: Skipped 1 previous similar message [13604.466619] Lustre: server umount tst250fs-MDT0001 complete [13610.290828] Lustre: server umount tst250fs-OST0000 complete [13615.788811] Lustre: server umount tst250fs-OST0001 complete [13640.000267] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [13650.572653] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [13690.432772] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:29:16 (1787747356) [13692.953796] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13716.563933] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [13733.510432] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [13777.672591] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13491 sec ========= 08:30:43 (1787747443) [13780.026562] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:30:45 (1787747445) === [13784.155625] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:30:50 (1787747450) === [13832.829343] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13837.290452] Key type lgssc unregistered [13837.924865] LNet: 385993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13837.938905] LNetError: 385993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13837.976356] LNet: Removed LNI 192.168.203.119@tcp [13839.448157] Key type .llcrypt unregistered [13839.450984] Key type ._llcrypt unregistered