[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 471638038 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002413] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007555] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011091] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013040] SELinux: Initializing. [ 0.014096] *** VALIDATE selinux *** [ 0.023353] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028341] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029178] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031127] *** VALIDATE tmpfs *** [ 0.032515] *** VALIDATE proc *** [ 0.034229] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039049] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043512] debug: unmapping init [mem 0xffffffffaee59000-0xffffffffaee60fff] [ 0.045200] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046690] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047028] ... version: 2 [ 0.048016] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051019] ... max period: 00007fffffffffff [ 0.052019] ... fixed-purpose events: 3 [ 0.053020] ... event mask: 000000070000000f [ 0.054288] rcu: Hierarchical SRCU implementation. [ 0.056457] smp: Bringing up secondary CPUs ... [ 0.057595] x86: Booting SMP configuration: [ 0.058028] .... node #0, CPUs: #1 #2 #3 [ 0.062331] smp: Brought up 1 node, 4 CPUs [ 0.064022] smpboot: Max logical packages: 1 [ 0.065023] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.230812] node 0 deferred pages initialised in 162ms [ 0.234221] devtmpfs: initialized [ 0.235234] x86/mm: Memory block size: 128MB [ 0.239418] gcov: version magic: 0x41383552 [ 0.243052] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.246125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.249380] pinctrl core: initialized pinctrl subsystem [ 0.251243] [ 0.251640] ************************************************************* [ 0.253018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.256017] ** ** [ 0.258013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.260018] ** ** [ 0.263019] ** This means that this kernel is built to expose internal ** [ 0.265017] ** IOMMU data structures, which may compromise security on ** [ 0.267013] ** your system. ** [ 0.268013] ** ** [ 0.270018] ** If you see this message and you are not debugging the ** [ 0.273019] ** kernel, report this immediately to your vendor! ** [ 0.275015] ** ** [ 0.277018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280025] ************************************************************* [ 0.282458] NET: Registered protocol family 16 [ 0.284457] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.286153] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.288146] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.291497] cpuidle: using governor menu [ 0.293501] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.296593] PCI: Using configuration type 1 for base access [ 0.298155] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.309061] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.310021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.314115] cryptd: max_cpu_qlen set to 1000 [ 0.317266] ACPI: Added _OSI(Module Device) [ 0.319021] ACPI: Added _OSI(Processor Device) [ 0.320016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.322015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.327577] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.333374] ACPI: Interpreter enabled [ 0.334135] ACPI: PM: (supports S0 S3 S4 S5) [ 0.336014] ACPI: Using IOAPIC for interrupt routing [ 0.337144] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.340370] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.352226] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.354061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.359033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.362115] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.368631] acpiphp: Slot [2] registered [ 0.370142] acpiphp: Slot [5] registered [ 0.371101] acpiphp: Slot [6] registered [ 0.373163] acpiphp: Slot [7] registered [ 0.375159] acpiphp: Slot [8] registered [ 0.376151] acpiphp: Slot [9] registered [ 0.378165] acpiphp: Slot [10] registered [ 0.380153] acpiphp: Slot [3] registered [ 0.382245] acpiphp: Slot [4] registered [ 0.383137] acpiphp: Slot [11] registered [ 0.385213] acpiphp: Slot [12] registered [ 0.387188] acpiphp: Slot [13] registered [ 0.388125] acpiphp: Slot [14] registered [ 0.390134] acpiphp: Slot [15] registered [ 0.391168] acpiphp: Slot [16] registered [ 0.392101] acpiphp: Slot [17] registered [ 0.394136] acpiphp: Slot [18] registered [ 0.396164] acpiphp: Slot [19] registered [ 0.397124] acpiphp: Slot [20] registered [ 0.398137] acpiphp: Slot [21] registered [ 0.400165] acpiphp: Slot [22] registered [ 0.402218] acpiphp: Slot [23] registered [ 0.404095] acpiphp: Slot [24] registered [ 0.405134] acpiphp: Slot [25] registered [ 0.407135] acpiphp: Slot [26] registered [ 0.408150] acpiphp: Slot [27] registered [ 0.410131] acpiphp: Slot [28] registered [ 0.411128] acpiphp: Slot [29] registered [ 0.412132] acpiphp: Slot [30] registered [ 0.414318] acpiphp: Slot [31] registered [ 0.415078] PCI host bridge to bus 0000:00 [ 0.417030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.419032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.422036] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.424029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.426034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.428034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.429205] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.432121] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.436568] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.447926] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.453019] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.456027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.458026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.461023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.463686] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.466801] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.470059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.473663] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.478022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.491021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.497981] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.504597] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.517022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.523023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.541023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.550349] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.557024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.567046] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.587030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.595207] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.601031] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.607024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.631021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.648404] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.661024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.669023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.689023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.706759] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.719024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.726025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.741021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.750188] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.755024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.760022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.775024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.789156] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.792485] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.794524] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.797434] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.799264] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.805209] iommu: Default domain type: Passthrough [ 0.808435] SCSI subsystem initialized [ 0.810210] ACPI: bus type USB registered [ 0.811168] usbcore: registered new interface driver usbfs [ 0.813103] usbcore: registered new interface driver hub [ 0.815129] usbcore: registered new device driver usb [ 0.817226] pps_core: LinuxPPS API ver. 1 registered [ 0.819025] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.822090] PTP clock support registered [ 0.825005] EDAC MC: Ver: 3.0.0 [ 0.826489] PCI: Using ACPI for IRQ routing [ 0.829540] NetLabel: Initializing [ 0.830011] NetLabel: domain hash size = 128 [ 0.831011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.833093] NetLabel: unlabeled traffic allowed by default [ 0.834167] vgaarb: loaded [ 0.836058] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.837015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.843000] clocksource: Switched to clocksource kvm-clock [ 0.951294] VFS: Disk quotas dquot_6.6.0 [ 0.952522] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.954281] *** VALIDATE ramfs *** [ 0.955069] *** VALIDATE hugetlbfs *** [ 0.956265] pnp: PnP ACPI init [ 0.958808] pnp: PnP ACPI: found 6 devices [ 0.974898] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.977550] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.979289] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.980495] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.982748] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.985140] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.986773] NET: Registered protocol family 2 [ 0.988602] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.993416] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.997166] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.003469] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.006867] TCP: Hash tables configured (established 65536 bind 65536) [ 1.009852] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.013303] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.016463] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.019716] NET: Registered protocol family 1 [ 1.023097] RPC: Registered named UNIX socket transport module. [ 1.025327] RPC: Registered udp transport module. [ 1.027138] RPC: Registered tcp transport module. [ 1.029038] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.031544] NET: Registered protocol family 44 [ 1.033287] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.035599] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.037656] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.040042] PCI: CLS 0 bytes, default 64 [ 1.041970] Unpacking initramfs... [ 2.487439] debug: unmapping init [mem 0xffff92e0bcc54000-0xffff92e0bffbffff] [ 2.491047] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.492688] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.494809] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 3.017356] Initialise system trusted keyrings [ 3.019055] Key type blacklist registered [ 3.020860] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.029892] zbud: loaded [ 3.032849] *** VALIDATE nfs *** [ 3.034270] *** VALIDATE nfs4 *** [ 3.035873] pstore: using deflate compression [ 3.039660] Platform Keyring initialized [ 3.159589] NET: Registered protocol family 38 [ 3.161137] Key type asymmetric registered [ 3.162591] Asymmetric key parser 'x509' registered [ 3.164803] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.168386] io scheduler mq-deadline registered [ 3.170149] io scheduler kyber registered [ 3.171834] io scheduler bfq registered [ 3.174245] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.177280] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.180280] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.183338] ACPI: Power Button [PWRF] [ 3.190066] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.197537] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.214366] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.222044] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.237043] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.264674] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.293904] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.298852] Non-volatile memory driver v1.3 [ 3.300199] Linux agpgart interface v0.103 [ 3.332465] virtio_blk virtio1: [vda] 145168 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.335669] vda: detected capacity change from 0 to 74326016 [ 3.356696] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.359382] vdb: detected capacity change from 0 to 1073741824 [ 3.378069] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.381097] vdc: detected capacity change from 0 to 2621440000 [ 3.395356] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.398878] vdd: detected capacity change from 0 to 2621440000 [ 3.416323] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.419292] vde: detected capacity change from 0 to 4294967296 [ 3.434810] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.437032] vdf: detected capacity change from 0 to 4294967296 [ 3.445535] libphy: Fixed MDIO Bus: probed [ 3.454543] usbcore: registered new interface driver usbserial_generic [ 3.456394] usbserial: USB Serial support registered for generic [ 3.458931] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.462969] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.464855] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.467633] mousedev: PS/2 mouse device common for all mice [ 3.470663] rtc_cmos 00:05: RTC can wake from S4 [ 3.471877] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.475784] rtc_cmos 00:05: registered as rtc0 [ 3.478131] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.478361] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.481482] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.482382] intel_pstate: CPU model not supported [ 3.487392] hid: raw HID events driver (C) Jiri Kosina [ 3.489042] usbcore: registered new interface driver usbhid [ 3.490472] usbhid: USB HID core driver [ 3.491661] drop_monitor: Initializing network drop monitor service [ 3.493412] Initializing XFRM netlink socket [ 3.495066] NET: Registered protocol family 10 [ 3.498606] Segment Routing with IPv6 [ 3.499666] NET: Registered protocol family 17 [ 3.501284] mpls_gso: MPLS GSO support [ 3.506091] RAS: Correctable Errors collector initialized. [ 3.507442] AVX version of gcm_enc/dec engaged. [ 3.508344] AES CTR mode by8 optimization enabled [ 3.579491] sched_clock: Marking stable (3579449635, 0)->(4541414849, -961965214) [ 3.582705] registered taskstats version 1 [ 3.584699] Loading compiled-in X.509 certificates [ 3.586343] zswap: loaded using pool lzo/zbud [ 3.614462] Key type big_key registered [ 3.626781] Key type encrypted registered [ 3.628492] ima: No TPM chip found, activating TPM-bypass! [ 3.632355] ima: Allocated hash algorithm: sha1 [ 3.633744] ima: No architecture policies found [ 3.635141] evm: Initialising EVM extended attributes: [ 3.636316] evm: security.selinux [ 3.637269] evm: security.ima [ 3.638165] evm: security.capability [ 3.638794] evm: HMAC attrs: 0x1 [ 3.640974] rtc_cmos 00:05: setting system clock to 2026-06-23 16:43:28 UTC (1782233008) [ 3.647479] debug: unmapping init [mem 0xffffffffafe03000-0xffffffffafffffff] [ 3.650155] debug: unmapping init [mem 0xffffffffaeb82000-0xffffffffaee58fff] [ 3.658109] Write protecting the kernel read-only data: 28672k [ 3.660417] debug: unmapping init [mem 0xffffffffad203000-0xffffffffad3fffff] [ 3.662379] debug: unmapping init [mem 0xffffffffadb14000-0xffffffffadbfffff] [ 3.695166] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.703707] systemd[1]: Detected virtualization kvm. [ 3.705205] systemd[1]: Detected architecture x86-64. [ 3.706579] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.729263] systemd[1]: No hostname configured. [ 3.730926] systemd[1]: Set hostname to . [ 3.733109] random: systemd: uninitialized urandom read (16 bytes read) [ 3.735069] systemd[1]: Initializing machine ID from random generator. [ 3.880062] random: systemd: uninitialized urandom read (16 bytes read) [ 3.882661] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.886423] random: systemd: uninitialized urandom read (16 bytes read) [ 3.889422] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.893928] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.467303] device-mapper: uevent: version 1.0.3 [ 4.468802] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.196780] virtio_net virtio0 ens2: renamed from eth0 [ 5.238522] random: fast init done [ 5.294272] scsi host0: ata_piix [ 5.324835] scsi host1: ata_piix [ 5.330340] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.333173] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.972071] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.143039] random: crng init done [ 10.144418] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.477880] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.565419] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.786764] SELinux: Disabled at runtime. [ 11.844893] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.852563] systemd[1]: Detected virtualization kvm. [ 11.854048] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.297914] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.301617] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.306349] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.311286] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.314826] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.321862] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.329245] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ 12.348884] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.734120] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.981322] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.048727] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.124498] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.145523] EDAC sbridge: Ver: 1.1.2 [ 14.476605] Key type dns_resolver registered [ 14.774935] NFS: Registering the id_resolver key type [ 14.776656] Key type id_resolver registered [ 14.778393] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg219-server login: [ 73.575936] libcfs: loading out-of-tree module taints kernel. [ 73.638622] Key type ._llcrypt registered [ 73.643371] Key type .llcrypt registered [ 73.738580] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 97.493698] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 100.092644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 100.124301] alg: No test for adler32 (adler32-zlib) [ 102.085372] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 103.116117] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 105.007212] Key type lgssc registered [ 107.944578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.297203] hrtimer: interrupt took 6180498 ns [ 130.095872] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 177.203989] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 192.853510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.912395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.248884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 194.331788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.477157] Lustre: lustre-MDT0000: new disk, initializing [ 194.658547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.702889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.625710] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.813043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.980147] Lustre: 6512:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.044162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.051559] Lustre: Skipped 1 previous similar message [ 216.176599] Lustre: lustre-MDT0001: new disk, initializing [ 216.293261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.336404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.349607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.234225] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.576747] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 236.274761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.577378] Lustre: lustre-OST0000: new disk, initializing [ 236.584860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 236.589375] Lustre: 8449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.689322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.191018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 238.202254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 238.277134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 243.888755] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.125912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.272652] Lustre: lustre-OST0001: new disk, initializing [ 259.277602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 259.283983] Lustre: 9522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.352849] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 265.017808] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.393039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 268.410233] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 268.519992] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.814750] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 285.350119] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 292.699558] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing check_logdir /tmp/testlogs/ [ 300.103376] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing yml_node [ 304.405276] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 307.498777] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 310.770469] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 313.305907] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 23 12:48:34 EDT 2026 [ 335.577489] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 337.391657] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 354.784180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 354.790868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.805684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.809358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.814540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.844541] Lustre: Skipped 2 previous similar messages [ 360.934249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.949878] Lustre: Skipped 6 previous similar messages [ 361.237383] Lustre: server umount lustre-MDT0000 complete [ 365.198015] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782233370 with bad export cookie 5541131260436205449 [ 365.200053] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 365.211901] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 366.050736] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.058515] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.062967] Lustre: Skipped 1 previous similar message [ 370.081667] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.089701] Lustre: Skipped 1 previous similar message [ 371.818059] Lustre: server umount lustre-MDT0001 complete [ 375.970142] Lustre: server umount lustre-OST0000 complete [ 379.914081] Lustre: server umount lustre-OST0001 complete [ 386.268512] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 397.797135] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 436.299818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.609726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 436.651517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 436.767404] Lustre: lustre-MDT0000: new disk, initializing [ 436.893302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.917729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 442.002438] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 454.236872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.347782] Lustre: 15077:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 454.457716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 454.462261] Lustre: Skipped 1 previous similar message [ 454.550911] Lustre: lustre-MDT0001: new disk, initializing [ 454.667527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 454.718763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 454.736364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 459.291896] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 469.224408] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 475.278958] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 482.464811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.808928] Lustre: lustre-OST0000: new disk, initializing [ 482.816453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 482.829200] Lustre: 17047:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 482.941453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.839071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 484.851308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 485.027841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 489.580477] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 500.917824] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 506.216319] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 506.455323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 511.531371] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 511.753332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 515.554461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.563130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.577646] Lustre: Skipped 1 previous similar message [ 528.351364] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 528.606600] Lustre: server umount lustre-OST0000 complete [ 536.033436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.060327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.069670] Lustre: Skipped 6 previous similar messages [ 547.295166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 547.604868] Lustre: server umount lustre-MDT0000 complete [ 551.393375] LustreError: 16447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.425402] LustreError: 16447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 551.910267] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782233556 with bad export cookie 5541131260436206982 [ 551.918733] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.930609] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 552.383443] Lustre: server umount lustre-MDT0001 complete [ 559.036519] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:52:41 (1782233561) [ 579.704823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.240325] LustreError: 19925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.304719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.699862] LustreError: 19926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.125331] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 590.822879] LustreError: 19925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.762516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.953244] LustreError: 19926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.806325] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 610.279048] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 617.156334] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 619.717475] Lustre: Failing over lustre-MDT0000 [ 620.113041] Lustre: server umount lustre-MDT0000 complete [ 621.794627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.802414] LustreError: 19922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.830753] Lustre: Skipped 2 previous similar messages [ 638.048094] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782233626/real 1782233626] req@ffff92e13f9d2300 x1868806862571776/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782233642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 638.078460] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.093526] LustreError: 19925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.116933] LustreError: 19925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 640.040913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.631974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.636460] Lustre: Skipped 1 previous similar message [ 647.698672] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 652.780123] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 652.787449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 652.821532] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 653.478668] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 662.996565] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 668.129974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 668.140879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 668.163170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 668.165793] Lustre: Skipped 4 previous similar messages [ 673.313683] Lustre: server umount lustre-MDT0000 complete [ 676.324862] LustreError: 22305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 676.351792] LustreError: 22305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 676.714684] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782233681 with bad export cookie 5541131260436208396 [ 676.716373] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 676.723307] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 677.033240] Lustre: server umount lustre-MDT0001 complete [ 686.454934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.226284] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 700.859528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 706.382352] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 714.718650] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 721.052336] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 723.335112] Lustre: Failing over lustre-MDT0000 [ 723.863589] Lustre: server umount lustre-MDT0000 complete [ 726.436067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 726.451511] Lustre: Skipped 2 previous similar messages [ 741.858721] LustreError: 24595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.878460] LustreError: 24595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 742.879410] Lustre: 3645:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782233731/real 1782233731] req@ffff92e104786a00 x1868806862601344/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782233747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 742.901480] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 743.509568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.120775] LustreError: 3643:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e104786a00 x1868806862604288/t0(0) o250->MGC192.168.202.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 753.570222] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 758.064449] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 758.761686] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 758.770873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 758.782444] Lustre: Skipped 1 previous similar message [ 758.806315] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 769.949971] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 774.112653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 774.125464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.134913] Lustre: Skipped 3 previous similar messages [ 780.268436] Lustre: server umount lustre-MDT0000 complete [ 784.450342] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782233789 with bad export cookie 5541131260436209334 [ 784.470533] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 784.471705] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 784.769549] Lustre: server umount lustre-MDT0001 complete [ 795.049847] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:56:37 (1782233797) [ 812.482281] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 824.238709] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 868.848142] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 880.711827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.038783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 881.081240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 881.240494] Lustre: lustre-MDT0000: new disk, initializing [ 881.378930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.381939] Lustre: Skipped 3 previous similar messages [ 881.410693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 886.689365] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 897.463599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 897.641431] Lustre: 29639:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 897.970753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 902.596758] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 912.265598] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 918.618846] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 925.491063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 925.755567] Lustre: 31609:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 927.380312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 927.422990] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 927.565966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 927.582690] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 932.710336] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 942.283754] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 949.094147] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 973.793023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 973.795814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.802653] LustreError: Skipped 1 previous similar message [ 973.803437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 973.803442] Lustre: Skipped 2 previous similar messages [ 973.836798] Lustre: Skipped 3 previous similar messages [ 977.587949] Lustre: server umount lustre-OST0000 complete [ 987.883759] Lustre: server umount lustre-MDT0000 complete [ 989.163324] LustreError: 29652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.192028] LustreError: 29652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 992.592945] LustreError: 29632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782233997 with bad export cookie 5541131260436209740 [ 992.602374] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 992.616161] LustreError: 29632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 993.094196] Lustre: server umount lustre-MDT0001 complete [ 1004.591956] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1006.718453] Key type lgssc unregistered [ 1007.106844] LNet: 33591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.125679] LNetError: 33591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.149579] LNet: Removed LNI 192.168.202.119@tcp [ 1008.128348] Key type .llcrypt unregistered [ 1008.132896] Key type ._llcrypt unregistered [ 1038.700479] Key type ._llcrypt registered [ 1038.701837] Key type .llcrypt registered [ 1038.762704] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1039.766451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1039.883846] alg: No test for adler32 (adler32-zlib) [ 1041.185191] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 1041.549878] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1043.295567] Key type lgssc registered [ 1044.892693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1052.936120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1052.966216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.507147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1059.073942] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1064.930440] LustreError: 35010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.940870] LustreError: 35010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1069.327588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.690294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1074.952562] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1083.576441] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1090.526244] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1098.957822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1099.216809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.233863] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1114.639623] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1114.761513] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1134.791708] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:02:16 (1782234136) [ 1138.533842] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1141.607804] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1150.946768] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:02:32 (1782234152) [ 1163.761638] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:02:45 (1782234165) [ 1165.201064] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1174.199081] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1174.209328] Lustre: Skipped 2 previous similar messages [ 1176.236109] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:02:58 (1782234178) [ 1178.459921] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1178.470982] Lustre: Skipped 2 previous similar messages [ 1187.335163] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1187.340708] Lustre: Skipped 4 previous similar messages [ 1202.914673] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:03:25 (1782234205) [ 1205.995180] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1209.503923] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1220.445892] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:03:42 (1782234222) [ 1223.244671] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1223.253197] Lustre: Skipped 6 previous similar messages [ 1239.060475] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:04:01 (1782234241) [ 1257.960634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1257.962069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.967991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1258.005049] Lustre: Skipped 1 previous similar message [ 1262.455411] Lustre: server umount lustre-OST0000 complete [ 1268.192246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.209314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1268.222648] Lustre: Skipped 1 previous similar message [ 1268.232032] Lustre: Skipped 1 previous similar message [ 1273.280970] Lustre: server umount lustre-MDT0000 complete [ 1273.320573] LustreError: 35010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.400833] LustreError: 35010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1277.211269] LustreError: 35758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782234282 with bad export cookie 2855708253144918171 [ 1277.221659] LustreError: 35758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1277.224021] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.471128] Lustre: server umount lustre-MDT0001 complete [ 1289.952630] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1292.694837] Key type lgssc unregistered [ 1293.021753] LNet: 42880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.027809] LNetError: 42880:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.043948] LNet: Removed LNI 192.168.202.119@tcp [ 1294.017924] Key type .llcrypt unregistered [ 1294.026224] Key type ._llcrypt unregistered [ 1314.279401] Key type ._llcrypt registered [ 1314.282522] Key type .llcrypt registered [ 1314.425239] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 1331.526431] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1332.457608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.480158] alg: No test for adler32 (adler32-zlib) [ 1333.713533] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 1334.019756] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1335.703190] Key type lgssc registered [ 1336.968452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1376.901767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1376.921694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.169325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1378.208536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1378.318185] Lustre: lustre-MDT0000: new disk, initializing [ 1378.381049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.404231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1383.158787] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1395.129440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.293174] Lustre: 46036:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1395.427949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1395.433190] Lustre: Skipped 1 previous similar message [ 1395.534812] Lustre: lustre-MDT0001: new disk, initializing [ 1395.722934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1395.810287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1395.840251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1401.287425] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1412.476179] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1418.854936] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1425.640479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1425.909870] Lustre: lustre-OST0000: new disk, initializing [ 1425.913478] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1425.925366] Lustre: 48001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1426.028226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.160804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1432.187074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1432.313931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1432.948783] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1443.516575] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1448.633819] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1448.916864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1454.225964] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1454.480923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1458.149369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.157875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1458.183273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1458.191581] Lustre: Skipped 1 previous similar message [ 1463.069129] Lustre: server umount lustre-OST0000 complete [ 1468.388944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1468.390790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1468.402867] Lustre: Skipped 2 previous similar messages [ 1468.418156] Lustre: Skipped 1 previous similar message [ 1473.510483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1473.969638] Lustre: server umount lustre-MDT0000 complete [ 1477.952637] LustreError: 48000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782234482 with bad export cookie 1754203302018232641 [ 1477.957396] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1477.970298] LustreError: 48000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1478.233350] Lustre: server umount lustre-MDT0001 complete [ 1489.380109] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:08:11 (1782234491) [ 1496.253982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.664317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.924805] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1501.672963] LustreError: 50463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.697922] LustreError: 50463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1506.787177] LustreError: 50462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.040413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.536306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1518.414641] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1529.260555] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1536.861915] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1545.920230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1546.397321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1554.906187] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1564.557936] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1574.507912] LustreError: 52992:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1575.583887] Lustre: Modifying parameter general.timeout=1 in log params [ 1576.875546] Lustre: Modifying parameter general.timeout=2 in log params [ 1579.042509] Lustre: Modifying parameter general.timeout=4 in log params [ 1579.045672] Lustre: Skipped 1 previous similar message [ 1583.746678] Lustre: Modifying parameter general.timeout=9 in log params [ 1583.756668] Lustre: Skipped 4 previous similar messages [ 1592.459203] Lustre: Modifying parameter general.timeout=18 in log params [ 1592.463780] Lustre: Skipped 8 previous similar messages [ 1608.787789] Lustre: Modifying parameter general.timeout=32 in log params [ 1608.793896] Lustre: Skipped 13 previous similar messages [ 1641.618373] Lustre: Modifying parameter general.timeout=64 in log params [ 1641.620410] Lustre: Skipped 31 previous similar messages [ 1715.709777] Lustre: Disabling parameter general.timeout= in log params [ 1715.712236] Lustre: Skipped 65 previous similar messages [ 1719.174942] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:12:01 (1782234721) [ 1722.267863] Lustre: 59945:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1724.399408] Lustre: 60054:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1726.420098] Lustre: 60154:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1732.033438] Lustre: 60468:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1732.043975] Lustre: 60468:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 1760.294548] Lustre: 61829:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1760.317226] LustreError: 61829:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1765.300852] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:12:47 (1782234767) [ 1779.684278] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.709543] Lustre: Skipped 1 previous similar message [ 1779.713643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1779.730285] Lustre: Skipped 3 previous similar messages [ 1784.802626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1784.820029] Lustre: Skipped 2 previous similar messages [ 1785.689580] Lustre: server umount lustre-MDT0000 complete [ 1789.394785] LustreError: 50444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782234794 with bad export cookie 1754203302018233530 [ 1789.403067] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1789.406882] LustreError: 50444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1789.730874] Lustre: server umount lustre-MDT0001 complete [ 1803.607749] Lustre: server umount lustre-OST0000 complete [ 1832.242573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.455960] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1832.660117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1837.555988] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1847.212500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.279314] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1847.299662] Lustre: 64367:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1852.380712] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1861.091528] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1867.047217] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1874.898640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1875.197396] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1875.647124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.657133] Lustre: Skipped 1 previous similar message [ 1882.572688] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1890.592718] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1892.499240] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1892.509655] Lustre: Skipped 23 previous similar messages [ 1892.551107] Lustre: 66932:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1903.584467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1903.598516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.609884] Lustre: Skipped 1 previous similar message [ 1903.618149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1906.664249] Lustre: server umount lustre-OST0000 complete [ 1914.344523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1914.362339] LustreError: Skipped 1 previous similar message [ 1914.364933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.402661] Lustre: Skipped 2 previous similar messages [ 1916.968089] Lustre: server umount lustre-MDT0000 complete [ 1919.459118] LustreError: 64378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.489263] LustreError: 64378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1921.771064] LustreError: 64359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782234926 with bad export cookie 1754203302018236701 [ 1921.775457] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1921.786074] LustreError: 64359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1922.190574] Lustre: server umount lustre-MDT0001 complete [ 1934.999838] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1938.471758] Key type lgssc unregistered [ 1938.777203] LNet: 68147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.783059] LNetError: 68147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.799162] LNet: Removed LNI 192.168.202.119@tcp [ 1939.780211] Key type .llcrypt unregistered [ 1939.782089] Key type ._llcrypt unregistered [ 1970.173391] Key type ._llcrypt registered [ 1970.174895] Key type .llcrypt registered [ 1970.335691] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1971.867628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1971.901204] alg: No test for adler32 (adler32-zlib) [ 1973.083919] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 1973.441978] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1975.152109] Key type lgssc registered [ 1976.679118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1985.444991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1985.480631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.079932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.862684] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1996.259734] LustreError: 69563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.272438] LustreError: 69563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2001.398512] LustreError: 69562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.666736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.379215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2007.146685] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2015.487467] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2022.179482] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2029.607183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2030.189659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.387962] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2046.922505] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2055.698285] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2067.425990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.431184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2067.451327] Lustre: Skipped 1 previous similar message [ 2067.468061] Lustre: Skipped 2 previous similar messages [ 2072.558929] LustreError: 69562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.587375] LustreError: 69562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2073.155784] Lustre: server umount lustre-MDT0000 complete [ 2077.665767] LustreError: 70572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.694647] LustreError: 70572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2077.847475] LustreError: 69544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782235082 with bad export cookie 7709106786411678225 [ 2077.858726] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2077.861988] LustreError: 69544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2078.256919] Lustre: server umount lustre-MDT0001 complete [ 2093.033463] Lustre: server umount lustre-OST0000 complete [ 2121.347749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.727642] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2121.756624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2122.074689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.912842] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2136.645497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.760614] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2136.786305] Lustre: 74432:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2136.895582] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2137.245307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2141.297951] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2150.698144] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2158.094152] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2167.066853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2167.182919] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2167.457829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.532443] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2183.898399] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2186.190411] Lustre: 77006:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2188.164403] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2196.960696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2196.977492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.992426] Lustre: Skipped 1 previous similar message [ 2196.996568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2201.227516] Lustre: server umount lustre-OST0000 complete [ 2207.712505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.740594] Lustre: Skipped 1 previous similar message [ 2207.747595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.755297] Lustre: Skipped 3 previous similar messages [ 2211.968423] Lustre: server umount lustre-MDT0000 complete [ 2212.837919] LustreError: 74444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.851799] LustreError: 74444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2216.282732] LustreError: 74423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782235221 with bad export cookie 7709106786411679065 [ 2216.287535] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2216.296690] LustreError: 74423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2216.773357] Lustre: server umount lustre-MDT0001 complete [ 2231.254580] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2234.522669] Key type lgssc unregistered [ 2234.696655] LNet: 78222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.703249] LNetError: 78222:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2235.758882] LNet: Removed LNI 192.168.202.119@tcp [ 2237.123707] Key type .llcrypt unregistered [ 2237.131729] Key type ._llcrypt unregistered [ 2273.062947] Key type ._llcrypt registered [ 2273.065661] Key type .llcrypt registered [ 2273.219856] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2274.445859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2274.575486] alg: No test for adler32 (adler32-zlib) [ 2275.722207] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 2275.970914] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2277.639734] Key type lgssc registered [ 2278.942800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2286.864601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2286.879781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.511984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2293.636990] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2298.864094] LustreError: 79638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.895277] LustreError: 79638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2303.730349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.968645] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2304.509149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2309.267456] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2319.301434] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2326.209622] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2334.910953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2335.620076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.270588] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2352.869544] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2365.938244] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2367.033850] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2369.009306] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2369.015945] Lustre: Skipped 1 previous similar message [ 2371.208326] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2371.214551] Lustre: Skipped 1 previous similar message [ 2375.469179] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2375.473851] Lustre: Skipped 3 previous similar messages [ 2384.098905] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2384.110401] Lustre: Skipped 7 previous similar messages [ 2400.314349] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2400.317062] Lustre: Skipped 14 previous similar messages [ 2432.648929] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2432.651638] Lustre: Skipped 28 previous similar messages [ 2485.688307] Lustre: 87770:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2489.053885] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:24:51 (1782235491) [ 2490.998756] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2492.870190] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:24:55 (1782235495) [ 2500.579156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2500.581077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2500.600985] Lustre: Skipped 1 previous similar message [ 2504.579359] Lustre: server umount lustre-OST0000 complete [ 2509.280541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2509.290311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.304866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2509.315080] Lustre: Skipped 1 previous similar message [ 2510.819177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.232727] Lustre: server umount lustre-MDT0000 complete [ 2515.936198] LustreError: 80371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.961989] LustreError: 80371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2519.407344] LustreError: 86166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782235524 with bad export cookie 6794968323945231585 [ 2519.413875] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2520.176489] Lustre: server umount lustre-MDT0001 complete [ 2534.367932] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2536.987757] Key type lgssc unregistered [ 2537.226735] LNet: 89023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2537.231887] LNetError: 89023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2538.277723] LNet: Removed LNI 192.168.202.119@tcp [ 2539.274772] Key type .llcrypt unregistered [ 2539.279260] Key type ._llcrypt unregistered [ 2544.875126] Key type ._llcrypt registered [ 2544.882357] Key type .llcrypt registered [ 2545.075545] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2549.483392] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules [ 2550.961768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.087245] alg: No test for adler32 (adler32-zlib) [ 2552.335338] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 2552.762202] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2554.567308] Key type lgssc registered [ 2556.224023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2557.933087] LustreError: 90234:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2578.183950] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2579.979629] LustreError: 90234:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2588.245160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2588.277739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.850439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2594.379898] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2599.392766] LustreError: 91251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.409788] LustreError: 91251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2604.529249] LustreError: 91252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.266777] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:26:47 (1782235607) [ 2609.645975] LustreError: 91251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.204124] LustreError: 91283:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2611.220252] LustreError: 91283:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2611.229749] LustreError: 91283:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2612.245882] Lustre: server umount lustre-MDT0000 complete [ 2632.730923] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2637.368748] Key type lgssc unregistered [ 2637.778856] LNet: 92779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2637.788588] LNetError: 92779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2637.820188] LNet: Removed LNI 192.168.202.119@tcp [ 2639.363591] Key type .llcrypt unregistered [ 2639.367513] Key type ._llcrypt unregistered [ 2663.009609] Key type ._llcrypt registered [ 2663.011743] Key type .llcrypt registered [ 2663.089652] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2664.146177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2664.402338] alg: No test for adler32 (adler32-zlib) [ 2665.570705] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 2665.903718] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2667.687620] Key type lgssc registered [ 2669.387583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2679.513646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2679.542253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.275262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2686.360533] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2690.543648] LustreError: 94000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.570105] LustreError: 94000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2695.648419] LustreError: 94001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.148482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2696.929373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2702.559371] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2712.109194] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2720.043222] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2727.855799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2728.649955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2735.298505] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2744.202418] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2810.579829] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:30:12 (1782235812) [ 2812.656488] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2815.539917] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:30:17 (1782235817) [ 2823.147753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.155111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2823.170706] Lustre: Skipped 2 previous similar messages [ 2823.194355] Lustre: Skipped 2 previous similar messages [ 2827.824791] Lustre: server umount lustre-MDT0000 complete [ 2828.269093] LustreError: 94731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.291093] LustreError: 94731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2833.378224] LustreError: 97288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.409315] LustreError: 97288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2833.843407] LustreError: 93982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782235838 with bad export cookie 13377638028843371755 [ 2833.858152] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2833.866963] LustreError: 93982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2834.452205] Lustre: server umount lustre-MDT0001 complete [ 2850.080245] Lustre: server umount lustre-OST0000 complete [ 2863.856701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.304925] LustreError: 98321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.591204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2869.994464] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2874.851022] LustreError: 98322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.881811] LustreError: 98322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2880.579989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.443429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2887.016668] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2897.032262] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2904.437113] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2920.973965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.382361] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2921.397686] LustreError: 98310:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2921.442642] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2921.455445] LustreError: 100407:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2921.468721] LustreError: 100407:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2921.482079] LustreError: 100407:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2921.603346] Lustre: server umount lustre-OST0000 complete [ 2921.611979] LustreError: 100407:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2933.629361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2933.793581] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2933.800656] LustreError: 98310:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2933.822831] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2933.832425] LustreError: 100874:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2933.837824] LustreError: 100874:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2933.843846] LustreError: 100874:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2933.929516] Lustre: server umount lustre-OST0000 complete [ 2933.934232] LustreError: 100874:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2947.163244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.439704] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2947.459247] Lustre: Found index 0 for lustre-OST0000, updating log [ 2947.468044] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2947.622867] Lustre: lustre-OST0000: new disk, initializing [ 2947.632717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2947.644202] Lustre: 101409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2948.195109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2956.016643] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2960.624165] Lustre: Failing over lustre-OST0000 [ 2960.791032] Lustre: server umount lustre-OST0000 complete [ 2968.033301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2968.044317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.067918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2969.569088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.570208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2969.593007] Lustre: Skipped 1 previous similar message [ 2974.688886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.698409] Lustre: Skipped 1 previous similar message [ 2975.834926] Lustre: server umount lustre-MDT0000 complete [ 2979.818687] LustreError: 98317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.841587] LustreError: 98317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2980.683139] LustreError: 98303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782235985 with bad export cookie 13377638028843374499 [ 2980.692285] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.704799] LustreError: 98303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2991.383091] Lustre: server umount lustre-MDT0001 complete [ 3003.149089] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:33:24 (1782236004) [ 3026.484370] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3041.759393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.245810] LustreError: 103758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.384692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.935484] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3062.654786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3071.503562] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3076.059129] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3086.340933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3087.319392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.331246] Lustre: Skipped 1 previous similar message [ 3087.359561] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3088.316402] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3088.380145] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3088.383329] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3088.403680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3096.502251] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3108.040393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3108.129745] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3108.226455] Lustre: lustre-OST0001: new disk, initializing [ 3108.231183] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3108.236548] Lustre: 106078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3113.671519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3113.705601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3113.833577] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3116.592589] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3127.950737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3136.879899] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3144.166993] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3144.175495] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.202547] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3149.212489] Lustre: server umount lustre-MDT0001 complete [ 3149.795304] LustreError: 105488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.826136] LustreError: 105488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3164.969782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.212706] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3165.222545] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3165.300241] Lustre: 103747:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3166.272705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3166.285107] Lustre: Skipped 1 previous similar message [ 3166.570390] LustreError: 107608:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3166.580509] LustreError: 107608:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3166.597649] LustreError: 103750:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3166.785605] LustreError: 107624:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.119@tcp: cfg command failed: rc = -17 [ 3166.795539] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3166.795539] [ 3171.333792] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3171.342140] Lustre: Skipped 1 previous similar message [ 3172.551874] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3187.019518] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:36:28 (1782236188) [ 3214.108744] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3243.523355] Lustre: 109703:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3275.781264] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3281.976418] Lustre: 110839:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3390.956624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3390.978182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3391.006034] Lustre: Skipped 2 previous similar messages [ 3391.008415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3391.013764] Lustre: Skipped 3 previous similar messages [ 3391.401679] Lustre: server umount lustre-MDT0000 complete [ 3391.466321] LustreError: 103753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3391.518249] LustreError: 103753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 3396.435236] LustreError: 103741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782236401 with bad export cookie 13377638028843375507 [ 3396.442167] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3396.448776] LustreError: 103741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3397.022630] Lustre: server umount lustre-MDT0001 complete [ 3404.535643] Lustre: server umount lustre-OST0000 complete [ 3409.772695] Lustre: server umount lustre-OST0001 complete [ 3417.090937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3431.827167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3442.638613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3456.312933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3483.632469] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3497.414625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.432804] Lustre: lustre-MDT0000: reset Object Index mappings [ 3498.178112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3506.129204] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3520.698441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.747253] Lustre: lustre-MDT0001: reset Object Index mappings [ 3528.066616] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3532.828188] Lustre: 115962:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3543.619387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3546.724759] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3554.082719] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3565.756232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3567.473905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3567.652227] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3567.806342] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3575.741768] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3587.416769] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3591.997423] Lustre: 117840:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3659.067311] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:44:20 (1782236660) [ 3685.348430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3685.352686] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.360301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3685.360321] Lustre: Skipped 2 previous similar messages [ 3685.428373] Lustre: Skipped 8 previous similar messages [ 3691.088414] Lustre: server umount lustre-MDT0000 complete [ 3695.617889] LustreError: 114807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3695.648544] LustreError: 114807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 3697.085908] LustreError: 114790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782236701 with bad export cookie 13377638028843422659 [ 3697.095984] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3697.109613] LustreError: 114790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3704.010520] Lustre: server umount lustre-MDT0001 complete [ 3715.725858] Lustre: server umount lustre-OST0000 complete [ 3727.421290] Lustre: server umount lustre-OST0001 complete [ 3735.810940] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3748.740561] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3797.561060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.143739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3798.198536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3798.366346] Lustre: lustre-MDT0000: new disk, initializing [ 3798.529678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.554337] Lustre: Skipped 3 previous similar messages [ 3798.585396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3804.268923] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3813.735570] Lustre: server umount lustre-MDT0000 complete [ 3826.252738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3826.671970] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3826.708463] Lustre: 122114:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3826.731683] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3833.137366] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3839.378729] Lustre: server umount lustre-MDT0000 complete [ 3850.631431] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:47:32 (1782236852) [ 3852.659640] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3855.606549] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:47:37 (1782236857) [ 3880.886591] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3894.781437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3901.809849] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3914.180988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.411402] Lustre: 124194:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3914.515729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3914.521707] Lustre: Skipped 1 previous similar message [ 3914.649320] Lustre: lustre-MDT0001: new disk, initializing [ 3914.884640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3914.899576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3922.059642] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3928.185420] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3937.326659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.801343] Lustre: lustre-OST0000: new disk, initializing [ 3937.807331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3937.824904] Lustre: 125776:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3939.775055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3939.790667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3940.015686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3947.636642] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3963.592469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3963.978938] Lustre: lustre-OST0001: new disk, initializing [ 3963.991261] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3964.012130] Lustre: 126649:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3966.129476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3966.156753] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3966.411759] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3973.366769] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3987.443411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3993.167823] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4401.990399] Lustre: 133063:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4402.000170] Lustre: 133063:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4402.006481] Lustre: 133063:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4402.010409] Lustre: 133063:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4402.021590] Lustre: 133063:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4402.037788] Lustre: 133063:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4709.670209] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:01:51 (1782237711) [ 4727.276716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4727.282715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4727.308907] Lustre: Skipped 4 previous similar messages [ 4727.335921] Lustre: Skipped 14 previous similar messages [ 4732.395855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4732.409251] Lustre: Skipped 3 previous similar messages [ 4732.963070] Lustre: server umount lustre-MDT0000 complete [ 4737.094508] LustreError: 133086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782237741 with bad export cookie 13377638028843493821 [ 4737.100922] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4737.110938] LustreError: 133086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4737.505078] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4737.511765] LustreError: 125701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4737.523791] Lustre: Skipped 2 previous similar messages [ 4737.564655] LustreError: 125701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4742.625426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4742.635823] Lustre: Skipped 3 previous similar messages [ 4744.224301] Lustre: server umount lustre-MDT0001 complete [ 4755.588979] Lustre: server umount lustre-OST0000 complete [ 4775.391641] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4775.640698] Lustre: server umount lustre-OST0001 complete [ 4783.608727] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4799.960328] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4838.947781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4839.315559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4839.388576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4839.494404] Lustre: lustre-MDT0000: new disk, initializing [ 4839.566579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.573753] Lustre: Skipped 5 previous similar messages [ 4839.590061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4844.277260] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4855.346418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.434359] Lustre: 136155:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4855.484942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4855.489182] Lustre: Skipped 1 previous similar message [ 4855.596311] Lustre: lustre-MDT0001: new disk, initializing [ 4855.702196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4855.718343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4860.514476] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4870.753540] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4876.740055] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4884.480921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4884.749848] Lustre: lustre-OST0000: new disk, initializing [ 4884.754202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4884.762771] Lustre: 138125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4886.670779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4886.679222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4886.841090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4893.768783] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4905.247766] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4917.572472] Lustre: lustre-MDD0000: changelog on [ 4920.767929] Lustre: lustre-MDD0001: changelog on [ 6912.359341] Lustre: lustre-MDD0001: changelog off [ 6914.404679] Lustre: 136164:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6915.254318] Lustre: lustre-MDD0000: changelog off [ 6924.774628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6924.774962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6924.782660] LustreError: Skipped 1 previous similar message [ 6924.797050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6924.809376] Lustre: Skipped 1 previous similar message [ 6927.234557] Lustre: server umount lustre-OST0000 complete [ 6934.497650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6934.514824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6934.538900] Lustre: Skipped 1 previous similar message [ 6934.541627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6938.221647] Lustre: server umount lustre-MDT0000 complete [ 6940.142832] LustreError: 136163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6940.167953] LustreError: 136163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6942.933103] LustreError: 136150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782239947 with bad export cookie 13377638028844314522 [ 6942.937397] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6942.955102] LustreError: 136150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6943.375481] Lustre: server umount lustre-MDT0001 complete [ 6958.943834] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6962.590570] Key type lgssc unregistered [ 6962.971363] LNet: 147277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6962.986250] LNetError: 147277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6963.009090] LNet: Removed LNI 192.168.202.119@tcp [ 6964.162158] Key type .llcrypt unregistered [ 6964.163476] Key type ._llcrypt unregistered [ 6989.853091] Key type ._llcrypt registered [ 6989.856034] Key type .llcrypt registered [ 6989.950716] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7008.524424] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7009.707036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7009.789176] alg: No test for adler32 (adler32-zlib) [ 7010.928473] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 7011.176752] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7012.887804] Key type lgssc registered [ 7013.936961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7052.465895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7052.524615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.830875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7053.861207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7053.950139] Lustre: lustre-MDT0000: new disk, initializing [ 7054.027725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7054.046747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7059.438837] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7071.044901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7071.214777] Lustre: 150483:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7071.270296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7071.279706] Lustre: Skipped 1 previous similar message [ 7071.355940] Lustre: lustre-MDT0001: new disk, initializing [ 7071.454804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7071.496721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7071.516139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7075.857321] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7085.093518] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7090.683307] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7097.165072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7097.417838] Lustre: lustre-OST0000: new disk, initializing [ 7097.421869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7097.430982] Lustre: 152453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7097.528310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7103.251242] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7105.599845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7105.611393] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7105.763910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7112.777685] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7119.531717] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_module obdecho/obdecho [ 7121.434713] LustreError: 153457:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7121.446673] LustreError: 153457:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7123.180950] LustreError: 153554:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 7136.743361] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.744533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7136.758084] Lustre: Skipped 2 previous similar messages [ 7136.766159] Lustre: Skipped 2 previous similar messages [ 7141.438462] Lustre: server umount lustre-MDT0000 complete [ 7144.822098] LustreError: 150474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782240149 with bad export cookie 10179087178564957568 [ 7144.833617] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7144.835551] LustreError: 150474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7145.153616] Lustre: server umount lustre-MDT0001 complete [ 7158.956959] Lustre: server umount lustre-OST0000 complete [ 7181.194299] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7192.982029] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7240.295624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.530673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7240.559619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7240.670281] Lustre: lustre-MDT0000: new disk, initializing [ 7240.762307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7240.791077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7245.447932] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7257.271074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7257.413906] Lustre: 157269:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7257.516547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7257.527480] Lustre: Skipped 1 previous similar message [ 7257.654127] Lustre: lustre-MDT0001: new disk, initializing [ 7257.769604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7257.817733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7257.841375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7262.887533] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7272.945497] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7278.948748] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7286.706366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7286.949820] Lustre: lustre-OST0000: new disk, initializing [ 7286.953115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7286.958969] Lustre: 159237:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7287.056338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7293.725329] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7295.021363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7295.034038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7295.145953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7304.420405] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7310.215447] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7310.550501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7316.073495] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7316.277310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7321.061299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7321.062635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7321.077672] LustreError: Skipped 1 previous similar message [ 7321.079402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7321.110098] Lustre: Skipped 1 previous similar message [ 7324.759886] Lustre: server umount lustre-OST0000 complete [ 7329.762241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7329.766955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7329.776109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7329.779019] Lustre: Skipped 1 previous similar message [ 7335.475932] Lustre: server umount lustre-MDT0000 complete [ 7336.424311] LustreError: 157281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.452337] LustreError: 157281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7339.228896] LustreError: 157262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782240344 with bad export cookie 10179087178564958527 [ 7339.234019] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7339.250905] LustreError: 157262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7339.734229] Lustre: server umount lustre-MDT0001 complete [ 7343.870798] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:45:45 (1782240345) [ 7346.302783] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7348.493621] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:45:50 (1782240350) [ 7358.541485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7358.899500] LustreError: 161653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7358.954123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7364.086255] LustreError: 161654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7364.487626] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7368.225451] LustreError: 161653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7373.281126] LustreError: 161654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7373.930946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7379.328849] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7387.846374] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7396.240815] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7404.286361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7404.763824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.778396] Lustre: Skipped 1 previous similar message [ 7411.871079] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7420.325036] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7438.827779] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.846365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.855394] Lustre: Skipped 2 previous similar messages [ 7438.873359] Lustre: Skipped 4 previous similar messages [ 7443.514781] Lustre: server umount lustre-MDT0000 complete [ 7443.938963] LustreError: 161650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.970863] LustreError: 161650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7447.307526] LustreError: 161634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782240452 with bad export cookie 10179087178564959409 [ 7447.310888] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7447.318814] LustreError: 161634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7449.064738] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7449.098797] Lustre: Skipped 1 previous similar message [ 7449.106187] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7449.112782] Lustre: Skipped 1 previous similar message [ 7453.785722] Lustre: server umount lustre-MDT0001 complete [ 7462.357190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7477.434521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.730073] LustreError: 165527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7477.743889] LustreError: 165527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7477.855800] LustreError: 165522:0:(llog.c:1646:llog_backup()) MGC192.168.202.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7477.861767] Lustre: 165522:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7513.568855] LustreError: 165529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7513.591614] LustreError: 165529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 7538.655428] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7538.669390] Lustre: 165522:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.202.119@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7578.083197] LustreError: 165527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.107309] LustreError: 165527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 7579.648489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7584.752769] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7586.123921] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7596.390572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7608.306627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7617.505194] LustreError: 148703:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007e96d258 x1868814107154688/t0(0) o250->MGC192.168.202.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7623.068892] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7623.176441] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7631.928211] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7633.646792] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7640.105436] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7642.383383] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7664.102646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7664.106722] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7664.114309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7664.114319] Lustre: Skipped 2 previous similar messages [ 7664.158861] Lustre: Skipped 2 previous similar messages [ 7664.367672] Lustre: server umount lustre-MDT0000 complete [ 7668.076153] LustreError: 161635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782240672 with bad export cookie 10179087178564965898 [ 7668.078720] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7668.090102] LustreError: 161635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7668.451673] Lustre: server umount lustre-MDT0001 complete [ 7683.720940] Lustre: server umount lustre-OST0000 complete [ 7708.188420] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7720.737460] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7759.301225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7759.568777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7759.625932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7759.763854] Lustre: lustre-MDT0000: new disk, initializing [ 7759.888704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7759.895803] Lustre: Skipped 1 previous similar message [ 7759.918308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7764.770680] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7776.482781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7776.769746] Lustre: 170635:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7777.227358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7783.549686] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7794.607170] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7801.430926] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7808.848680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7809.141867] Lustre: lustre-OST0000: new disk, initializing [ 7809.145313] Lustre: Skipped 1 previous similar message [ 7809.154608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7809.163763] Lustre: Skipped 2 previous similar messages [ 7809.173650] Lustre: 172602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7815.806514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7815.815870] Lustre: Skipped 1 previous similar message [ 7815.826513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7816.018273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7817.622934] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7828.421186] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7832.944424] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7833.175719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7838.250827] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7838.471415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7841.762241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7841.762566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7841.786138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7841.794555] LustreError: Skipped 1 previous similar message [ 7841.804403] Lustre: Skipped 3 previous similar messages [ 7846.770047] Lustre: server umount lustre-OST0000 complete [ 7857.458961] Lustre: server umount lustre-MDT0000 complete [ 7860.974755] LustreError: 170626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782240865 with bad export cookie 10179087178564982698 [ 7860.982481] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7860.987964] LustreError: 170626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7861.588398] Lustre: server umount lustre-MDT0001 complete [ 7871.598924] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:54:33 (1782240873) [ 7879.724959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7879.979387] LustreError: 175014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7879.997726] LustreError: 175014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 7884.761583] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7894.338449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7900.818641] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7910.497383] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7917.843514] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7926.474940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7934.485199] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7942.449893] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7965.286559] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:56:07 (1782240967) [ 7970.278195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7970.279967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7970.282120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7970.282129] Lustre: Skipped 4 previous similar messages [ 7970.297417] Lustre: Skipped 4 previous similar messages [ 7970.307402] LustreError: Skipped 1 previous similar message [ 7976.270940] Lustre: server umount lustre-OST0000 complete [ 7994.335585] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7994.707654] Lustre: server umount lustre-MDT0000 complete [ 7998.829865] LustreError: 174994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241003 with bad export cookie 10179087178564983580 [ 7998.854840] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7999.062180] Lustre: server umount lustre-MDT0001 complete [ 8012.706100] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8016.256904] Key type lgssc unregistered [ 8016.526763] LNet: 178768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8016.531247] LNetError: 178768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8016.551389] LNet: Removed LNI 192.168.202.119@tcp [ 8017.382332] Key type .llcrypt unregistered [ 8017.384435] Key type ._llcrypt unregistered [ 8039.288984] Key type ._llcrypt registered [ 8039.292225] Key type .llcrypt registered [ 8039.398507] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8040.958847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8040.991790] alg: No test for adler32 (adler32-zlib) [ 8042.137588] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 8042.496665] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8044.186509] Key type lgssc registered [ 8045.187791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8053.467100] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8053.480383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8069.026438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8074.143350] LustreError: 180012:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [ 8079.264453] LustreError: 180012:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8079.284931] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8079.299213] LustreError: MGC192.168.202.119@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8079.320446] LustreError: 180012:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8079.351417] LustreError: 180012:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8079.375960] LustreError: 180012:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8079.460964] Lustre: server umount lustre-OST0000 complete [ 8079.466545] LustreError: 180012:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8089.118422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8090.748100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8095.862704] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8100.328837] LustreError: 180512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8100.348092] LustreError: 180512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8105.283202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.443303] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8105.606850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8110.845070] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8119.904587] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8127.511664] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8134.808981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8135.090262] LustreError: 182329:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8135.110228] Lustre: 182329:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.119@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8135.292049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8142.689353] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8151.700987] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8156.131617] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8156.140282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8156.148149] Lustre: Skipped 1 previous similar message [ 8156.157282] Lustre: Skipped 2 previous similar messages [ 8161.108243] Lustre: server umount lustre-MDT0000 complete [ 8161.255332] LustreError: 180513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8161.274570] LustreError: 180513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8165.386742] LustreError: 182357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241170 with bad export cookie 7293138076183128353 [ 8165.394564] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8165.399460] LustreError: 182357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8165.855654] Lustre: server umount lustre-MDT0001 complete [ 8179.577361] Lustre: server umount lustre-OST0000 complete [ 8191.408496] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8204.045622] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8247.773271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8248.189587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8248.254958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8248.395495] Lustre: lustre-MDT0000: new disk, initializing [ 8248.547203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8248.570982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8252.908966] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8264.341418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8264.532833] Lustre: 185855:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8264.723681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8264.733617] Lustre: Skipped 1 previous similar message [ 8264.935359] Lustre: lustre-MDT0001: new disk, initializing [ 8265.110894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8265.182238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8265.197351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8270.212519] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8280.274264] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8286.767349] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8295.529850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8295.817517] Lustre: lustre-OST0000: new disk, initializing [ 8295.822286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8295.829793] Lustre: 187835:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8295.920164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8302.864826] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8304.172748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8304.191743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8304.297568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8313.236243] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8318.568286] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8318.897859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8323.877907] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8324.156649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8330.213623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8330.217731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8330.228826] LustreError: Skipped 1 previous similar message [ 8330.235556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8330.257096] Lustre: Skipped 2 previous similar messages [ 8330.284260] Lustre: Skipped 1 previous similar message [ 8332.559418] Lustre: server umount lustre-OST0000 complete [ 8340.448730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.450468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8340.472870] Lustre: Skipped 1 previous similar message [ 8343.295779] Lustre: server umount lustre-MDT0000 complete [ 8345.568286] LustreError: 187844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8345.591848] LustreError: 187844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8347.348549] LustreError: 187834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241352 with bad export cookie 7293138076183129067 [ 8347.361557] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8347.375863] LustreError: 187834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8347.953579] Lustre: server umount lustre-MDT0001 complete [ 8359.402813] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:02:41 (1782241361) [ 8382.257709] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8385.132241] Key type lgssc unregistered [ 8385.449781] LNet: 190782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8385.458828] LNetError: 190782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8385.475286] LNet: Removed LNI 192.168.202.119@tcp [ 8386.559871] Key type .llcrypt unregistered [ 8386.564279] Key type ._llcrypt unregistered [ 8406.808603] Key type ._llcrypt registered [ 8406.812730] Key type .llcrypt registered [ 8406.929703] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8407.952102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8407.975918] alg: No test for adler32 (adler32-zlib) [ 8409.005639] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 8409.258889] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8410.920681] Key type lgssc registered [ 8412.471859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8421.425390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8421.476622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8423.114296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8428.131745] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8432.624093] LustreError: 192002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8432.645394] LustreError: 192002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8437.664236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.728159] LustreError: 192001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8438.130823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8443.354131] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8451.125502] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8457.318405] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8466.749251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8467.022225] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8467.025694] LustreError: 193866:0:(llog.c:1687:llog_backup()) MGC192.168.202.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8467.037610] Lustre: 193866:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.119@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8467.168176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8473.565656] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8481.378090] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8484.320281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8484.336106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8484.358510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8487.909921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.912316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8487.933225] Lustre: Skipped 1 previous similar message [ 8487.946152] Lustre: Skipped 2 previous similar messages [ 8490.216951] Lustre: server umount lustre-MDT0000 complete [ 8493.034259] LustreError: 191996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8493.060077] LustreError: 191996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8493.867737] LustreError: 191982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241498 with bad export cookie 3096078928676662592 [ 8493.879025] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8493.887899] LustreError: 191982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8494.305492] Lustre: server umount lustre-MDT0001 complete [ 8509.208774] Lustre: server umount lustre-OST0000 complete [ 8522.721491] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8534.112813] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8575.047376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8575.313231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8575.420170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8575.549973] Lustre: lustre-MDT0000: new disk, initializing [ 8575.654694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8575.689294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8581.031484] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8594.958101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8595.100538] Lustre: 197391:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8595.188239] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8595.190128] Lustre: Skipped 1 previous similar message [ 8595.193523] LustreError: 198175:0:(llog.c:1687:llog_backup()) MGC192.168.202.119@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8595.200819] LustreError: 198175:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8595.204437] Lustre: 198175:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.119@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8595.226043] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8595.229516] Lustre: Skipped 1 previous similar message [ 8595.360385] Lustre: lustre-MDT0001: new disk, initializing [ 8595.494738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8595.549944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8595.570418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8601.638532] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8612.023456] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8618.792423] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8625.967936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8626.153827] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8626.156180] Lustre: Skipped 1 previous similar message [ 8626.158615] LustreError: 199338:0:(llog.c:1687:llog_backup()) MGC192.168.202.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8626.162953] LustreError: 199338:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8626.169680] Lustre: 199338:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.119@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8626.173529] Lustre: 199338:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8626.190172] Lustre: lustre-OST0000: new disk, initializing [ 8626.193366] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8626.201877] Lustre: 199363:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8626.291970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8631.925061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8631.931466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8632.026672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8633.139892] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8643.867722] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8650.575613] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8650.896162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8656.127745] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8656.364417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8663.015973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8663.018185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8663.022703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8663.062127] Lustre: Skipped 1 previous similar message [ 8664.626934] Lustre: server umount lustre-OST0000 complete [ 8672.225078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8672.236796] LustreError: Skipped 1 previous similar message [ 8672.250101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8672.273890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8672.278613] Lustre: Skipped 1 previous similar message [ 8675.277434] Lustre: server umount lustre-MDT0000 complete [ 8678.373703] LustreError: 197398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8678.406057] LustreError: 197398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8679.895513] LustreError: 197381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241684 with bad export cookie 3096078928676663313 [ 8679.897339] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8679.921818] LustreError: 197381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8680.332524] Lustre: server umount lustre-MDT0001 complete [ 8690.503768] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:08:12 (1782241692) [ 8712.864230] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8728.435509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.912610] LustreError: 202510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8728.998623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8734.166387] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8738.278581] LustreError: 202511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8738.329323] LustreError: 202511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8745.265222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8750.506803] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8754.127792] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8762.303950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8762.712971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8762.722134] Lustre: Skipped 1 previous similar message [ 8770.350397] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8780.458929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8780.710578] Lustre: lustre-OST0001: new disk, initializing [ 8780.717357] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8780.736368] Lustre: 204820:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8787.536760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8787.556594] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8787.707793] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8787.947722] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8801.044749] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8809.670514] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8812.258028] Lustre: DEBUG MARKER: ADD OST3 [ 8819.649432] loop: module loaded [ 8819.750927] loop0: detected capacity change from 0 to 204800000 [ 8820.076936] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8820.095104] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8820.115458] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8828.298289] loop0: detected capacity change from 0 to 204800000 [ 8828.806821] Lustre: DEBUG MARKER: STOP OST3 [ 8834.458084] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8834.679089] Lustre: lustre-OST0003: new disk, initializing [ 8834.681874] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8834.687716] Lustre: 206856:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8834.808324] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8834.817725] Lustre: Skipped 1 previous similar message [ 8836.129317] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8836.311536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8836.318882] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8836.355782] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8838.197631] LustreError: 206982:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8838.343306] Lustre: Failing over lustre-OST0003 [ 8838.543663] Lustre: server umount lustre-OST0003 complete [ 8841.715605] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8841.728582] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8841.739522] Lustre: Skipped 1 previous similar message [ 8841.740470] LustreError: 203993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8841.740477] LustreError: 203993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8842.156892] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8852.313658] Lustre: DEBUG MARKER: START OST3 again [ 8859.162527] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8859.565170] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8861.427114] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8861.593848] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8861.594259] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8867.274478] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8892.790604] Lustre: Failing over lustre-OST0003 [ 8893.031543] Lustre: server umount lustre-OST0003 complete [ 8895.486654] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8895.523771] Lustre: Skipped 1 previous similar message [ 8895.540143] LustreError: 203993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8895.577156] LustreError: 203993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8914.920681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8914.929630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8914.937634] Lustre: Skipped 1 previous similar message [ 8914.942204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8914.947616] Lustre: Skipped 2 previous similar messages [ 8920.114516] Lustre: server umount lustre-MDT0000 complete [ 8925.255957] LustreError: 202493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241930 with bad export cookie 3096078928676664195 [ 8925.270592] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8925.300360] LustreError: 202493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8926.185095] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8926.200933] Lustre: Skipped 5 previous similar messages [ 8932.076761] Lustre: server umount lustre-MDT0001 complete [ 8936.255316] Lustre: server umount lustre-OST0000 complete [ 8939.960990] Lustre: server umount lustre-OST0001 complete [ 8947.464814] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8958.034851] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8999.486128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8999.723424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8999.753407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8999.821588] Lustre: lustre-MDT0000: new disk, initializing [ 8999.883972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8999.888660] Lustre: Skipped 1 previous similar message [ 8999.898650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9004.253376] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9015.927713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.045742] Lustre: 212288:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9016.092907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9016.096528] Lustre: Skipped 1 previous similar message [ 9016.164736] Lustre: lustre-MDT0001: new disk, initializing [ 9016.275061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9016.286297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9021.736550] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9031.179858] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9038.059642] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9046.399303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9046.739658] Lustre: 214256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9048.223539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9048.529301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9055.215255] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9065.230272] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9070.212549] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9070.401614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9077.287777] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9077.713314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9084.384949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9084.402555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9084.405396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9084.422278] Lustre: Skipped 6 previous similar messages [ 9084.450170] Lustre: Skipped 3 previous similar messages [ 9086.396362] Lustre: server umount lustre-OST0000 complete [ 9105.376051] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9105.865964] Lustre: server umount lustre-MDT0000 complete [ 9109.990849] LustreError: 212299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9110.014989] LustreError: 212299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 9110.192388] LustreError: 212280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242115 with bad export cookie 3096078928676666491 [ 9110.196309] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9110.209807] LustreError: 212280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9110.952545] Lustre: server umount lustre-MDT0001 complete [ 9115.065492] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 15:15:17 (1782242117) [ 9132.845839] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 9145.178574] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9185.492510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9185.853301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9185.890135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9185.895174] Lustre: Skipped 1 previous similar message [ 9185.956760] Lustre: lustre-MDT0000: new disk, initializing [ 9185.962130] Lustre: Skipped 1 previous similar message [ 9186.044674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9186.050653] Lustre: Skipped 1 previous similar message [ 9191.623155] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9204.103409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9204.312093] Lustre: 218870:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9204.637125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9209.996123] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9221.673391] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9227.808604] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9234.856898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9235.226668] Lustre: 220842:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9237.174359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9237.348961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9242.929501] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9253.700956] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9258.663239] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9258.875646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9264.176409] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9264.406600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9268.198193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9268.201546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9268.204298] LustreError: Skipped 3 previous similar messages [ 9268.213772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9268.220146] Lustre: Skipped 3 previous similar messages [ 9268.232553] Lustre: Skipped 8 previous similar messages [ 9273.219671] Lustre: server umount lustre-OST0000 complete [ 9283.834367] Lustre: server umount lustre-MDT0000 complete [ 9288.259416] LustreError: 218860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242293 with bad export cookie 3096078928676667373 [ 9288.267630] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9288.616754] Lustre: server umount lustre-MDT0001 complete [ 9296.732044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9297.300497] LustreError: 223060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9297.409761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9297.418034] Lustre: Skipped 5 previous similar messages [ 9302.409632] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9311.795866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9316.692730] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9325.298509] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9331.678819] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9338.233375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9344.799151] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9352.922820] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9367.015663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 9367.037199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9367.040094] Lustre: Skipped 4 previous similar messages [ 9370.112314] Lustre: server umount lustre-OST0000 complete [ 9381.558683] Lustre: server umount lustre-MDT0000 complete [ 9386.130685] LustreError: 223040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242390 with bad export cookie 3096078928676668255 [ 9386.137240] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9386.147068] LustreError: 223040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9386.724932] Lustre: server umount lustre-MDT0001 complete [ 9494.009467] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 9496.637810] Key type lgssc unregistered [ 9496.973641] LNet: 226722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9496.988378] LNetError: 226722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9497.005220] LNet: Removed LNI 192.168.202.119@tcp [ 9497.745247] Key type .llcrypt unregistered [ 9497.749613] Key type ._llcrypt unregistered [ 9529.261653] Key type ._llcrypt registered [ 9529.264439] Key type .llcrypt registered [ 9529.381902] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 9545.365552] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9546.439229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9546.528581] alg: No test for adler32 (adler32-zlib) [ 9547.538524] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 9547.801555] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 9549.431205] Key type lgssc registered [ 9550.632612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9589.686794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9589.737591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9591.011661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9591.041387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9591.160513] Lustre: lustre-MDT0000: new disk, initializing [ 9591.384202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9591.412499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9595.685319] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9606.186148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9606.405524] Lustre: 230092:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9606.456752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9606.460618] Lustre: Skipped 1 previous similar message [ 9606.491369] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9606.515523] Lustre: lustre-MDT0001: new disk, initializing [ 9606.610687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9606.637629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9606.649963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9611.809646] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9622.354401] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9630.099572] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9638.509643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9638.732336] Lustre: lustre-OST0000: new disk, initializing [ 9638.736305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9638.742084] Lustre: 232060:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9638.833647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9644.645449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9644.662603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9644.850837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9645.340571] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9655.515597] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9660.461316] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9660.666337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9665.322911] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9665.551300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9670.624249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9670.629151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9670.641539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9670.658380] Lustre: Skipped 1 previous similar message [ 9673.842675] Lustre: server umount lustre-OST0000 complete [ 9678.304108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9678.312520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9678.322976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9678.327167] Lustre: Skipped 1 previous similar message [ 9680.867506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9680.883923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9683.617895] Lustre: server umount lustre-MDT0000 complete [ 9687.829540] LustreError: 230086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242692 with bad export cookie 11730364145301228856 [ 9687.832964] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9687.846895] LustreError: 230086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9688.233433] Lustre: server umount lustre-MDT0001 complete [ 9709.296262] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9720.830421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9721.282376] LustreError: 234995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9721.300523] LustreError: 234995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9721.334983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9726.294771] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9726.434202] LustreError: 234996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9731.557525] LustreError: 234995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9734.365549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9734.764822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9739.107124] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9741.871716] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9748.826028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9749.199708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9756.037960] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9765.486678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9765.640598] Lustre: lustre-OST0001: new disk, initializing [ 9765.643983] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9765.649324] Lustre: 237304:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9765.737327] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9773.146581] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9775.729860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9775.740597] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9775.826710] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9785.289901] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9796.456967] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9811.169251] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 15:26:53 (1782242813) [ 9816.551924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9816.565415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.591452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9816.596265] Lustre: Skipped 1 previous similar message [ 9820.222832] Lustre: server umount lustre-MDT0000 complete [ 9822.191184] LustreError: 234990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9822.221815] LustreError: 234990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9824.025174] LustreError: 234976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242828 with bad export cookie 11730364145301229738 [ 9824.031199] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9824.036063] LustreError: 234976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9824.324830] Lustre: server umount lustre-MDT0001 complete [ 9839.268387] Lustre: server umount lustre-OST0000 complete [ 9843.167692] Lustre: 228317:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782242832/real 1782242832] req@ffff92e10d411500 x1868816766321024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782242848 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9843.231693] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9843.242267] Lustre: Skipped 3 previous similar messages [ 9843.921937] Lustre: server umount lustre-OST0001 complete [ 9852.609708] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 9868.674189] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9910.854601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9911.083513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9911.112779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9911.246139] Lustre: lustre-MDT0000: new disk, initializing [ 9911.318814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9911.335542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9916.231752] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9927.196694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9927.333547] Lustre: 241271:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9927.395096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9927.400676] Lustre: Skipped 1 previous similar message [ 9927.469267] Lustre: lustre-MDT0001: new disk, initializing [ 9927.593733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9927.611217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9932.285870] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9943.122842] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9949.114967] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9955.878496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9956.287391] Lustre: lustre-OST0000: new disk, initializing [ 9956.291699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9956.311980] Lustre: 243240:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9958.013732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9958.022510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9958.149441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9963.774598] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9973.676559] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9979.334079] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9979.544493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9984.393876] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9984.631935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9989.088484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9989.091731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9989.122105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9989.143045] Lustre: Skipped 5 previous similar messages [ 9992.994942] Lustre: server umount lustre-OST0000 complete [ 9999.328849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10003.486350] Lustre: server umount lustre-MDT0000 complete [10004.447863] LustreError: 241283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10004.461184] LustreError: 241283:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10007.116833] LustreError: 242077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243011 with bad export cookie 11730364145301231873 [10007.121182] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10007.131264] LustreError: 242077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10007.481205] Lustre: server umount lustre-MDT0001 complete [10016.188381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10016.831965] LustreError: 245455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10016.922921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10016.932652] Lustre: Skipped 2 previous similar messages [10022.106365] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10031.302712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10037.648681] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10048.779989] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10054.931354] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10062.156288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10067.590617] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10074.915284] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10220.519667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10220.525652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10220.550030] Lustre: Skipped 3 previous similar messages [10220.556727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10220.557160] LustreError: Skipped 1 previous similar message [10220.606650] Lustre: Skipped 3 previous similar messages [10225.655888] LustreError: 247277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10225.669583] LustreError: 247277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10225.746238] Lustre: server umount lustre-OST0000 complete [10236.069748] Lustre: server umount lustre-MDT0000 complete [10240.219498] LustreError: 245437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243245 with bad export cookie 11730364145301232748 [10240.231830] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10240.564031] Lustre: server umount lustre-MDT0001 complete [10344.681952] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [10347.735037] Key type lgssc unregistered [10348.067621] LNet: 249065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10348.072799] LNetError: 249065:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10349.096959] LNet: Removed LNI 192.168.202.119@tcp [10350.013184] Key type .llcrypt unregistered [10350.016412] Key type ._llcrypt unregistered [10377.447281] Key type ._llcrypt registered [10377.450493] Key type .llcrypt registered [10377.654682] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [10394.385607] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10395.983175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10396.116727] alg: No test for adler32 (adler32-zlib) [10397.117505] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [10397.310059] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [10398.967226] Key type lgssc registered [10400.202270] Lustre: Echo OBD driver; http://www.lustre.org/ [10445.744290] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10456.843742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10456.868267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10458.195917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10458.230644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10458.342695] Lustre: lustre-MDT0000: new disk, initializing [10458.455690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10458.483174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10463.385656] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10474.611741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10474.765757] Lustre: 253135:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10474.881856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10474.890649] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10474.900558] Lustre: Skipped 1 previous similar message [10475.019412] Lustre: lustre-MDT0001: new disk, initializing [10475.072906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10475.108084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10475.121357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10479.651749] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10484.875698] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10492.605771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10492.991387] Lustre: lustre-OST0000: new disk, initializing [10492.996154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10493.010731] Lustre: 254767:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10493.120954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10493.930607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10493.941648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10494.067333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10499.916497] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10510.426268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10510.582538] Lustre: lustre-OST0001: new disk, initializing [10510.585927] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10510.590628] Lustre: 255638:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10510.666614] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10517.461938] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10520.240148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10520.252585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10520.316657] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10530.443363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10539.875604] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10551.780424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10551.797953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10551.814669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10556.386135] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.392628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10556.417027] Lustre: Skipped 2 previous similar messages [10556.428794] Lustre: Skipped 3 previous similar messages [10561.508061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10561.513066] Lustre: Skipped 2 previous similar messages [10566.111927] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10566.317826] Lustre: server umount lustre-MDT0000 complete [10570.411826] LustreError: 253129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243575 with bad export cookie 5958337695227111469 [10570.416476] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10570.427960] LustreError: 253129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10570.836883] Lustre: server umount lustre-MDT0001 complete [10585.133865] Lustre: server umount lustre-OST0000 complete [10588.127230] Lustre: 250642:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782243576/real 1782243576] req@ffff92e006994a80 x1868817657578240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782243592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10588.173282] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10589.650769] Lustre: server umount lustre-OST0001 complete [10602.809596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10615.104439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10627.172377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10639.953710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10662.352406] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10672.857700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10672.893409] Lustre: lustre-MDT0000: reset Object Index mappings [10673.228626] LustreError: 260650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10673.245658] LustreError: 260650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10673.353734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10678.257310] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10678.764512] LustreError: 260651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.884334] LustreError: 260650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10687.876050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10687.901646] Lustre: lustre-MDT0001: reset Object Index mappings [10688.372403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10692.995956] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10696.397362] Lustre: 261793:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10703.855263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10710.732879] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10713.517028] LustreError: 262145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.640634] LustreError: 262147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.674510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10718.678749] LustreError: 262147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10720.313370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10720.622614] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10720.626526] Lustre: Skipped 1 previous similar message [10725.911709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10727.169761] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10736.303490] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10740.706827] Lustre: 263663:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10746.337961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10746.356599] Lustre: Skipped 1 previous similar message [10746.360816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10746.369684] Lustre: Skipped 4 previous similar messages [10748.327902] Lustre: server umount lustre-MDT0000 complete [10751.463741] LustreError: 262428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10751.495107] LustreError: 262428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10752.377483] LustreError: 260632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243757 with bad export cookie 5958337695227118259 [10752.380208] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10752.403063] LustreError: 260632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10752.856062] Lustre: server umount lustre-MDT0001 complete [10767.710860] Lustre: server umount lustre-OST0000 complete [10783.424299] Lustre: server umount lustre-OST0001 complete [10805.604960] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:43:28 (1782243808) [10822.557777] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [10834.096550] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10874.852825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10875.136349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10875.205169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10875.334795] Lustre: lustre-MDT0000: new disk, initializing [10875.415971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10875.445634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10881.356961] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10894.700660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10894.844843] Lustre: 267920:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10894.898893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10894.902284] Lustre: Skipped 1 previous similar message [10894.972861] Lustre: lustre-MDT0001: new disk, initializing [10895.166122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10895.179247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10900.598470] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10911.233599] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10918.313432] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10926.314244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10926.639726] Lustre: lustre-OST0000: new disk, initializing [10926.642909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10926.653123] Lustre: 269891:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10928.083134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10928.103161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10928.372757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10935.480671] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10947.041198] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10952.735451] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10953.136538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10958.540711] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10958.920108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10963.937362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10963.954353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10963.966350] Lustre: Skipped 2 previous similar messages [10963.971985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10967.413564] Lustre: server umount lustre-OST0000 complete [10972.129325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10972.141688] LustreError: Skipped 1 previous similar message [10972.148410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10972.160694] Lustre: Skipped 1 previous similar message [10978.053305] Lustre: server umount lustre-MDT0000 complete [10979.808543] LustreError: 267931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10979.847953] LustreError: 267931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10982.685963] LustreError: 268717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243987 with bad export cookie 5958337695227120345 [10982.697650] LustreError: 268717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10982.699329] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10983.025778] Lustre: server umount lustre-MDT0001 complete [11006.577265] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11017.982958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11018.276955] LustreError: 272825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11018.320164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11018.324020] Lustre: Skipped 2 previous similar messages [11023.849335] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11033.087835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11039.211205] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11042.447576] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11050.292718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11057.755439] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11067.827419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11068.118762] Lustre: lustre-OST0001: new disk, initializing [11068.123975] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11068.131450] Lustre: 275135:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11070.259828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11070.286414] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11075.603687] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11075.759731] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11087.663917] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11091.790181] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11098.416621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11098.448464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11105.250742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11105.274767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11105.307384] Lustre: Skipped 1 previous similar message [11105.320766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11105.327971] Lustre: Skipped 5 previous similar messages [11111.395866] LustreError: 273554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11111.429383] LustreError: 273554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11111.658692] Lustre: server umount lustre-MDT0000 complete [11115.411849] LustreError: 272806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244120 with bad export cookie 5958337695227121234 [11115.421847] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11115.428073] LustreError: 272806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11115.768607] Lustre: server umount lustre-MDT0001 complete [11130.664720] Lustre: server umount lustre-OST0000 complete [11132.900568] Lustre: 250640:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782244121/real 1782244121] req@ffff92e013afb480 x1868817657739520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782244137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11136.359742] Lustre: server umount lustre-OST0001 complete [11157.215822] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11170.756421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11171.624411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11171.634627] Lustre: Skipped 3 previous similar messages [11176.702813] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11187.527327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11193.378638] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11196.745313] Lustre: 279330:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11204.973427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11212.065298] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11216.701424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11220.592836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11226.118059] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11227.228811] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11235.744099] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11239.661604] Lustre: 281209:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11245.564872] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11245.585953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11259.757436] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:51:01 (1782244261) [11277.283277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11277.296331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11277.315153] Lustre: Skipped 5 previous similar messages [11277.346766] Lustre: Skipped 7 previous similar messages [11283.417782] Lustre: server umount lustre-MDT0000 complete [11287.533299] LustreError: 278192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11287.563463] LustreError: 278192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [11288.413931] LustreError: 278172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244293 with bad export cookie 5958337695227127800 [11288.421340] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11288.428707] LustreError: 278172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11295.257052] Lustre: server umount lustre-MDT0001 complete [11314.143166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11314.426896] Lustre: server umount lustre-OST0000 complete [11319.186783] Lustre: server umount lustre-OST0001 complete [11326.797486] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [11339.361712] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11388.842660] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11401.517139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11402.023630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11402.086910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11402.242508] Lustre: lustre-MDT0000: new disk, initializing [11402.310972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11407.473446] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11421.180678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11421.434866] Lustre: 285472:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11421.980605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11427.506403] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11440.794430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11441.202867] Lustre: 287105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11441.347377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11441.349470] Lustre: Skipped 5 previous similar messages [11442.683532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11448.184915] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11459.167114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11459.430199] Lustre: 287978:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11461.029591] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11461.042087] Lustre: Skipped 1 previous similar message [11461.174679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11466.465184] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11477.567387] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11482.223876] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11482.228370] Lustre: Skipped 1 previous similar message [11502.801782] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:55:04 (1782244504) [11505.911710] Lustre: server umount lustre-MDT0000 complete [11507.170271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11507.190256] Lustre: Skipped 5 previous similar messages [11510.702099] LustreError: 287977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244515 with bad export cookie 5958337695227135458 [11510.711447] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11510.715271] LustreError: 287977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11511.236814] Lustre: server umount lustre-MDT0001 complete [11525.296249] Lustre: server umount lustre-OST0000 complete [11528.481376] Lustre: 250640:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782244517/real 1782244517] req@ffff92e00c5f4e00 x1868817657899520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782244533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11529.803878] Lustre: server umount lustre-OST0001 complete [11541.010826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11546.303963] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11547.106135] LustreError: 290222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11547.134275] LustreError: 290222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [11555.319748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11560.584112] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11569.297541] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11576.473199] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11584.565211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11591.902982] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11600.834786] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11618.462227] loop1: detected capacity change from 0 to 204800000 [11618.745704] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11618.760532] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11618.773099] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11624.919997] loop1: detected capacity change from 0 to 204800000 [11631.833706] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11631.973794] LustreError: 290210:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs4616: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11631.989132] LustreError: 290210:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs4616 [11631.994682] LustreError: 290210:0:(mgs_handler.c:517:mgs_target_reg()) fs4616-MDT0001: Error checking target index: rc = -13 [11632.006459] LustreError: fs4616-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11632.021354] LustreError: 293941:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11632.032798] LustreError: 293941:0:(tgt_mount.c:2129:server_put_super()) no obd fs4616-MDT0001 [11632.037292] LustreError: 293941:0:(tgt_mount.c:116:server_deregister_mount()) fs4616-MDT0001 not registered [11632.093378] Lustre: server umount fs4616-MDT0001 complete [11632.098708] LustreError: 293941:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11640.906944] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11640.985651] Lustre: Setting parameter fs4616-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs4616-MDT0001 [11641.047336] Lustre: srv-fs4616-MDT0001: No data found on store. Initialize space. [11641.050283] Lustre: Skipped 4 previous similar messages [11641.103557] Lustre: fs4616-MDT0001: new disk, initializing [11641.105974] Lustre: Skipped 3 previous similar messages [11641.204090] Lustre: srv-fs4616-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11641.210174] Lustre: 294407:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4616-MDT0001-mdtlov: can't create new llog: rc = -115 [11641.223317] LustreError: 294407:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4616-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11645.568829] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11653.271159] Lustre: server umount fs4616-MDT0001 complete [11666.919961] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11667.670284] loop1: detected capacity change from 0 to 204800000 [11673.923604] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11674.374230] Lustre: 295977:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4616-MDT0001-mdtlov: can't create new llog: rc = -115 [11674.383152] LustreError: 295977:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4616-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11679.487666] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11682.292129] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11684.883841] Lustre: server umount fs4616-MDT0001 complete [11697.640156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11697.642447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11697.669030] Lustre: Skipped 5 previous similar messages [11697.687524] Lustre: Skipped 9 previous similar messages [11700.874781] Lustre: server umount lustre-MDT0000 complete [11708.719257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11708.829842] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11713.525359] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11714.550054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11721.244714] loop1: detected capacity change from 0 to 204800000 [11727.659604] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11728.433436] Lustre: 298427:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4616-MDT0001-mdtlov: can't create new llog: rc = -115 [11728.447721] LustreError: 298427:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4616-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11733.802856] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11739.249826] Lustre: server umount fs4616-MDT0001 complete [11764.194993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11768.353926] Lustre: server umount lustre-OST0000 complete [11778.396715] Lustre: server umount lustre-MDT0000 complete [11782.040716] LustreError: 290202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244786 with bad export cookie 5958337695227138139 [11782.046449] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11782.057604] LustreError: 290202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11782.440657] Lustre: server umount lustre-MDT0001 complete [11793.657302] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11796.440448] Key type lgssc unregistered [11796.817564] LNet: 300771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11796.821739] LNetError: 300771:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11796.841240] LNet: Removed LNI 192.168.202.119@tcp [11797.757299] Key type .llcrypt unregistered [11797.759698] Key type ._llcrypt unregistered [11816.682064] Key type ._llcrypt registered [11816.686266] Key type .llcrypt registered [11816.765490] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11817.832364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11817.868066] alg: No test for adler32 (adler32-zlib) [11818.914820] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [11819.192028] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11820.919179] Key type lgssc registered [11822.020807] Lustre: Echo OBD driver; http://www.lustre.org/ [11828.825221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11828.839477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11830.244381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11833.919108] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11840.485316] LustreError: 302043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11840.526804] LustreError: 302043:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11841.642338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11842.223972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11845.977312] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11852.529648] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11860.241813] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11868.325616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11868.568795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11874.701272] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11875.772548] LustreError: 303882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11880.939244] LustreError: 303882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11882.094855] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11884.848596] LustreError: 303882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11890.176314] LustreError: 303882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11890.216405] LustreError: 303882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11900.396726] LustreError: 304300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11900.426081] LustreError: 304300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11905.297118] loop1: detected capacity change from 0 to 204800000 [11905.573975] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11905.607363] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11905.640882] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11911.961418] loop1: detected capacity change from 0 to 204800000 [11917.808367] LustreError: 304300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11917.835593] LustreError: 304300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [11919.273172] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11919.590674] Lustre: fs345-OST0000: new disk, initializing [11919.596984] Lustre: srv-fs345-OST0000: No data found on store. Initialize space. [11919.601949] Lustre: 305789:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11919.744161] Lustre: fs345-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11926.145708] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11936.198925] Lustre: server umount fs345-OST0000 complete [11951.003986] loop1: detected capacity change from 0 to 204800000 [11951.597405] LustreError: 306807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11951.618182] LustreError: 306807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [11953.647320] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11956.504861] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11956.841311] Lustre: fs345-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11962.297859] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11969.002281] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11971.593437] loop2: detected capacity change from 0 to 204800000 [11971.854540] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11971.885411] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11971.913191] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11979.107543] loop2: detected capacity change from 0 to 204800000 [11986.674744] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11986.792285] LustreError: fs345-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11986.807079] LustreError: fs345-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11986.825573] LustreError: 308829:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11986.839305] LustreError: 308829:0:(tgt_mount.c:2129:server_put_super()) no obd fs345-OST0001 [11986.845139] LustreError: 308829:0:(tgt_mount.c:116:server_deregister_mount()) fs345-OST0001 not registered [11986.891554] Lustre: server umount fs345-OST0001 complete [11986.893462] LustreError: 308829:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11999.315162] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11999.625493] Lustre: fs345-OST0001: new disk, initializing [11999.644794] Lustre: srv-fs345-OST0001: No data found on store. Initialize space. [11999.653232] Lustre: 309382:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11999.765354] Lustre: fs345-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12006.292585] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12015.181454] Lustre: server umount fs345-OST0000 complete [12018.151749] LustreError: 303883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.187887] LustreError: 303883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [12024.491674] Lustre: server umount fs345-OST0001 complete [12036.891845] loop1: detected capacity change from 0 to 204800000 [12043.015548] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12043.219271] Lustre: fs345-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12049.866243] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12058.461978] loop2: detected capacity change from 0 to 204800000 [12064.953281] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12071.186718] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12082.090275] Lustre: server umount fs345-OST0000 complete [12091.135954] Lustre: server umount fs345-OST0001 complete [12106.211899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12106.221107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12106.247990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12107.234282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12107.249040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12109.618401] Lustre: server umount lustre-OST0000 complete [12113.379906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12113.389591] LustreError: Skipped 2 previous similar messages [12113.395156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12113.424435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.430020] Lustre: Skipped 1 previous similar message [12117.475198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12117.475923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12119.402099] Lustre: server umount lustre-MDT0000 complete [12124.099930] LustreError: 309381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245128 with bad export cookie 12742706087210756951 [12124.113363] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12124.123371] LustreError: 309381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12124.944137] Lustre: server umount lustre-MDT0001 complete [12139.432725] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12142.391884] Key type lgssc unregistered [12142.854692] LNet: 315309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12142.862463] LNetError: 315309:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12142.884798] LNet: Removed LNI 192.168.202.119@tcp [12144.141170] Key type .llcrypt unregistered [12144.144171] Key type ._llcrypt unregistered [12179.675320] Key type ._llcrypt registered [12179.677384] Key type .llcrypt registered [12179.775665] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [12196.950837] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12198.214839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12198.247424] alg: No test for adler32 (adler32-zlib) [12199.317688] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [12199.573622] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12201.215170] Key type lgssc registered [12202.467960] Lustre: Echo OBD driver; http://www.lustre.org/ [12241.908914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12241.969205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12243.287631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12243.303923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12243.349315] Lustre: lustre-MDT0000: new disk, initializing [12243.400967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12243.411049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12247.246558] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12258.761439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.901782] Lustre: 319170:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12259.029316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12259.033404] Lustre: Skipped 1 previous similar message [12259.171340] Lustre: lustre-MDT0001: new disk, initializing [12259.331202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12259.391537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12259.406715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12264.796312] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12277.420853] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12284.057994] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12293.044473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12293.309258] Lustre: lustre-OST0000: new disk, initializing [12293.313754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12293.319058] Lustre: 321142:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12293.421653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12299.356794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12299.388624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12299.564860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12299.799729] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12310.383961] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12312.677609] Lustre: Mounted lustre-client [12313.800124] Lustre: Unmounted lustre-client [12325.344678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12325.350454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12325.362559] LustreError: Skipped 1 previous similar message [12325.375637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12327.915341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12327.926793] Lustre: Skipped 2 previous similar messages [12329.238472] Lustre: server umount lustre-OST0000 complete [12336.100114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12336.109640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12336.121333] Lustre: Skipped 1 previous similar message [12336.134708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12338.144567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12338.145457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12339.562580] Lustre: server umount lustre-MDT0000 complete [12343.264628] LustreError: 319182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12343.289874] LustreError: 319182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12343.359417] LustreError: 321141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245348 with bad export cookie 6864937519506918076 [12343.373367] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12343.380122] LustreError: 321141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12343.695652] Lustre: server umount lustre-MDT0001 complete [12356.614719] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12359.874165] Key type lgssc unregistered [12369.510754] LNet: 323304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12369.535982] LNetError: 323304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12445.358806] LNet: Removed LNI 192.168.202.119@tcp [12446.849679] Key type .llcrypt unregistered [12446.851886] Key type ._llcrypt unregistered [12469.571361] Key type ._llcrypt registered [12469.576746] Key type .llcrypt registered [12469.758071] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [12487.566917] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12488.364836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12488.389231] alg: No test for adler32 (adler32-zlib) [12489.693341] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [12489.894477] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12491.552197] Key type lgssc registered [12492.917845] Lustre: Echo OBD driver; http://www.lustre.org/ [12527.994587] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:12:09 (1782245529) [12535.388309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12535.432329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12536.715801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12536.746578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12536.818457] Lustre: lustre-MDT0000: new disk, initializing [12536.897111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12536.916459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12541.448237] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12552.798830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12553.047927] Lustre: 326530:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12553.102299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12553.109568] Lustre: Skipped 1 previous similar message [12553.186877] Lustre: lustre-MDT0001: new disk, initializing [12553.300733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12553.343383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12553.356556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12558.499790] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12567.908942] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12574.516861] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12581.862758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12582.262515] Lustre: lustre-OST0000: new disk, initializing [12582.265716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12582.270729] Lustre: 328498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12582.376745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12588.142336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12588.168858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12588.413144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12590.667439] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12601.581280] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12619.235148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12619.242461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12619.255539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12619.263886] Lustre: Skipped 1 previous similar message [12622.382884] Lustre: server umount lustre-OST0000 complete [12629.478594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12629.486366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12629.506214] Lustre: Skipped 2 previous similar messages [12629.532677] Lustre: Skipped 1 previous similar message [12634.444340] Lustre: server umount lustre-MDT0000 complete [12639.712189] LustreError: 326540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12639.756029] LustreError: 326540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12640.048354] LustreError: 326521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245644 with bad export cookie 17460667688966293625 [12640.060098] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12640.068673] LustreError: 326521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12640.956624] Lustre: server umount lustre-MDT0001 complete [12655.897082] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12659.056864] Key type lgssc unregistered [12659.354066] LNet: 330408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12659.359891] LNetError: 330408:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12659.381195] LNet: Removed LNI 192.168.202.119@tcp [12660.228746] Key type .llcrypt unregistered [12660.234221] Key type ._llcrypt unregistered [12686.179562] Key type ._llcrypt registered [12686.184715] Key type .llcrypt registered [12686.358910] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12687.652972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12687.689094] alg: No test for adler32 (adler32-zlib) [12689.011264] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [12689.242140] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12690.904811] Key type lgssc registered [12692.051728] Lustre: Echo OBD driver; http://www.lustre.org/ [12699.690591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12699.713730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.301465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12705.748819] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12711.907207] LustreError: 331682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12711.931629] LustreError: 331682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12714.901590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12715.538111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12721.299389] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12728.865366] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12734.806396] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12742.675549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12743.073363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12750.049263] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12757.787577] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12771.067862] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12773.002338] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12780.036852] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [12781.026410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12781.038981] Lustre: Skipped 2 previous similar messages [12781.040905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12781.050223] Lustre: Skipped 2 previous similar messages [12783.787291] Lustre: server umount lustre-MDT0000 complete [12785.133317] LustreError: 331678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12786.144958] LustreError: 331682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12786.156484] LustreError: 331682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12788.196554] LustreError: 331663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245793 with bad export cookie 4410784179877234589 [12788.210814] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12788.212304] LustreError: 331663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12790.238407] LustreError: 331677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12790.279500] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [12791.268411] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12793.312995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12793.321715] Lustre: Skipped 1 previous similar message [12794.560356] Lustre: server umount lustre-MDT0001 complete [12798.718244] Lustre: server umount lustre-OST0000 complete [12807.782315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12812.603254] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12815.477455] Lustre: 335866:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.119@tcp [12817.942170] Lustre: server umount MGS complete [12826.506926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12826.806946] LustreError: 336440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12826.821659] LustreError: 336440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12826.881853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12831.070716] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12835.040239] LustreError: 336435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12835.068930] LustreError: 336435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12838.674360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12838.925095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12842.701992] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12850.281516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12850.512569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12856.378934] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12863.428489] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12865.043145] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12871.279948] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12873.133960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12878.822223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12878.843885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12878.884253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12879.330975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12879.866630] Lustre: server umount lustre-OST0000 complete [12884.448470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12884.464255] Lustre: Skipped 1 previous similar message [12889.570587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12889.581813] Lustre: Skipped 5 previous similar messages [12898.271146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12898.508958] Lustre: server umount lustre-MDT0000 complete [12899.809476] LustreError: 336436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12899.837056] LustreError: 336436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12901.831589] LustreError: 336422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245906 with bad export cookie 4410784179877235429 [12901.839641] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12901.844370] LustreError: 336422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12902.099729] Lustre: server umount lustre-MDT0001 complete [12913.356477] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12915.835659] Key type lgssc unregistered [12916.103847] LNet: 339876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12916.111802] LNetError: 339876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12916.124962] LNet: Removed LNI 192.168.202.119@tcp [12916.988463] Key type .llcrypt unregistered [12916.992486] Key type ._llcrypt unregistered [12936.705702] Key type ._llcrypt registered [12936.709433] Key type .llcrypt registered [12936.793657] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [12959.177857] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12960.564393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12960.644299] alg: No test for adler32 (adler32-zlib) [12961.632687] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [12961.870746] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12963.551994] Key type lgssc registered [12964.776208] Lustre: Echo OBD driver; http://www.lustre.org/ [13005.290554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13005.315746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13006.670484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13006.715528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13006.867106] Lustre: lustre-MDT0000: new disk, initializing [13006.952672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13006.998587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13011.913063] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13022.899569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13023.058814] Lustre: 343041:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13023.123779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13023.130465] Lustre: Skipped 1 previous similar message [13023.212237] Lustre: lustre-MDT0001: new disk, initializing [13023.316895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13023.369687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13023.381950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13028.136606] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13037.934612] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13043.927614] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13051.232840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13051.421547] Lustre: lustre-OST0000: new disk, initializing [13051.424977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13051.430189] Lustre: 345006:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13051.517779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13057.577945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13057.589575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13057.708174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13058.072650] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13067.991738] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13072.582523] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13072.833524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13077.109146] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13077.335402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13083.104145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13083.109755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13083.123325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13083.622760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13083.645248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13083.670321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13088.737422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13088.741091] Lustre: Skipped 1 previous similar message [13093.344066] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13093.549338] Lustre: server umount lustre-OST0000 complete [13098.976653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13098.983056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13099.014195] Lustre: Skipped 1 previous similar message [13099.032277] Lustre: Skipped 1 previous similar message [13103.690321] Lustre: server umount lustre-MDT0000 complete [13107.594031] LustreError: 343032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782246112 with bad export cookie 5789845158727328095 [13107.603201] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13108.013245] Lustre: server umount lustre-MDT0001 complete [13117.458638] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:21:59 (1782246119) [13133.996691] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [13144.103150] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [13195.001148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13195.257820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13195.293845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13195.375920] Lustre: lustre-MDT0000: new disk, initializing [13195.566408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13195.634491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13201.130174] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13212.427260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13212.554896] Lustre: 349772:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13212.625879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13212.637785] Lustre: Skipped 1 previous similar message [13212.756689] Lustre: lustre-MDT0001: new disk, initializing [13212.879331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13212.917657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13212.931845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13217.159513] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13227.740342] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13234.904813] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13242.949428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13243.449505] Lustre: lustre-OST0000: new disk, initializing [13243.456653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13243.470653] Lustre: 351740:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13243.615833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13250.614731] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13252.701363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13252.716808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13252.855926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13261.008454] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13299.173226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13299.182547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13299.221080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13299.221197] LustreError: Skipped 1 previous similar message [13299.237552] Lustre: Skipped 1 previous similar message [13304.062359] Lustre: server umount lustre-OST0000 complete [13310.433330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13310.456331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13310.477027] Lustre: Skipped 1 previous similar message [13310.504100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13316.412693] Lustre: server umount lustre-MDT0000 complete [13319.648619] LustreError: 349783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13319.669110] LustreError: 349783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13320.793427] LustreError: 349766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782246325 with bad export cookie 5789845158727328977 [13320.794807] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13320.807415] LustreError: 349766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13321.240634] Lustre: server umount lustre-MDT0001 complete [13334.617405] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13337.923407] Key type lgssc unregistered [13338.402951] LNet: 353915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13338.415924] LNetError: 353915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13339.431781] LNet: Removed LNI 192.168.202.119@tcp [13340.685031] Key type .llcrypt unregistered [13340.687663] Key type ._llcrypt unregistered [13439.126431] Key type ._llcrypt registered [13439.131633] Key type .llcrypt registered [13439.279489] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13439.492319] Key type .llcrypt unregistered [13439.493671] Key type ._llcrypt unregistered [13519.354198] Key type ._llcrypt registered [13519.356196] Key type .llcrypt registered [13519.440744] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13519.697848] Key type .llcrypt unregistered [13519.700487] Key type ._llcrypt unregistered [13558.468234] Key type ._llcrypt registered [13558.469676] Key type .llcrypt registered [13558.513919] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13558.621499] Key type .llcrypt unregistered [13558.622663] Key type ._llcrypt unregistered [13590.544917] Key type ._llcrypt registered [13590.549437] Key type .llcrypt registered [13590.659351] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13591.123577] Key type .llcrypt unregistered [13591.129680] Key type ._llcrypt unregistered [13688.041678] Key type ._llcrypt registered [13688.050549] Key type .llcrypt registered [13688.229594] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13688.504562] Key type .llcrypt unregistered [13688.511575] Key type ._llcrypt unregistered [13723.532865] Key type ._llcrypt registered [13723.536891] Key type .llcrypt registered [13723.690687] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13723.932403] Key type .llcrypt unregistered [13723.934276] Key type ._llcrypt unregistered [13758.164976] Key type ._llcrypt registered [13758.167913] Key type .llcrypt registered [13758.282497] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13758.563347] Key type .llcrypt unregistered [13758.565146] Key type ._llcrypt unregistered [13791.563792] Key type ._llcrypt registered [13791.566751] Key type .llcrypt registered [13791.657634] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13791.974850] Key type .llcrypt unregistered [13791.975973] Key type ._llcrypt unregistered [13823.675695] Key type ._llcrypt registered [13823.678370] Key type .llcrypt registered [13823.777629] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13824.076618] Key type .llcrypt unregistered [13824.081922] Key type ._llcrypt unregistered [13853.281894] Key type ._llcrypt registered [13853.283856] Key type .llcrypt registered [13853.375120] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13853.758368] Key type .llcrypt unregistered [13853.763956] Key type ._llcrypt unregistered [13920.858403] Key type ._llcrypt registered [13920.859740] Key type .llcrypt registered [13920.949570] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13921.288402] Key type .llcrypt unregistered [13921.290527] Key type ._llcrypt unregistered [13954.100274] Key type ._llcrypt registered [13954.104935] Key type .llcrypt registered [13954.268450] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13954.567939] Key type .llcrypt unregistered [13954.571449] Key type ._llcrypt unregistered [13996.000443] Key type ._llcrypt registered [13996.002934] Key type .llcrypt registered [13996.100200] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13996.343495] Key type .llcrypt unregistered [13996.345901] Key type ._llcrypt unregistered [14063.867153] Key type ._llcrypt registered [14063.870526] Key type .llcrypt registered [14064.031406] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [14084.664766] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [14085.989011] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14086.004463] alg: No test for adler32 (adler32-zlib) [14087.101170] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [14087.436883] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [14089.119230] Key type lgssc registered [14090.400143] Lustre: Echo OBD driver; http://www.lustre.org/ [14129.894250] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14129.926855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14131.235644] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14131.284726] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14131.380225] Lustre: tst250fs-MDT0000: new disk, initializing [14131.451796] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14131.473572] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14136.014701] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14148.857946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14149.032971] Lustre: 374263:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14149.102329] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14149.106419] Lustre: Skipped 1 previous similar message [14149.214952] Lustre: tst250fs-MDT0001: new disk, initializing [14149.381763] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14149.413284] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14149.425147] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14154.377463] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14159.613455] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14167.108334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14167.504992] Lustre: tst250fs-OST0000: new disk, initializing [14167.512630] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14167.533505] Lustre: 375895:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14167.667708] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14169.535023] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14169.552862] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14169.807356] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14174.807205] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14186.522954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14186.801689] Lustre: tst250fs-OST0001: new disk, initializing [14186.806841] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14186.817285] Lustre: 376766:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14186.962887] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14194.331211] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [14196.330629] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14196.339528] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14196.411630] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14200.951427] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14203.639249] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14203.642759] Lustre: Skipped 2 previous similar messages [14226.401170] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14226.410946] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14226.434508] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14227.429613] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14227.434558] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14227.471975] Lustre: Skipped 2 previous similar messages [14227.505642] Lustre: Skipped 3 previous similar messages [14228.982267] Lustre: server umount tst250fs-MDT0000 complete [14232.767747] LustreError: 374255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782247237 with bad export cookie 13097987759856833426 [14232.769693] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14232.784267] LustreError: 374255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14233.136953] Lustre: server umount tst250fs-MDT0001 complete [14248.007763] Lustre: server umount tst250fs-OST0000 complete [14262.313327] Lustre: server umount tst250fs-OST0001 complete [14282.506689] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [14291.768983] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [14327.665907] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:42:09 (1782247329) [14329.558870] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14349.136722] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [14359.908972] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [14394.927924] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14080 sec ========= 16:43:17 (1782247397) [14397.062866] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:43:19 (1782247399) === [14401.207566] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:43:23 (1782247403) === [14449.084583] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [14452.253684] Key type lgssc unregistered [14452.682840] LNet: 385903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14452.693968] LNetError: 385903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14452.715401] LNet: Removed LNI 192.168.202.119@tcp [14453.697146] Key type .llcrypt unregistered [14453.700569] Key type ._llcrypt unregistered