[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 742974777 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003082] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010004] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011016] pid_max: default: 32768 minimum: 301 [ 0.012136] LSM: Security Framework initializing [ 0.013069] Yama: becoming mindful. [ 0.015013] SELinux: Initializing. [ 0.016097] *** VALIDATE selinux *** [ 0.030615] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.037000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.037145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.040084] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.043047] *** VALIDATE tmpfs *** [ 0.044459] *** VALIDATE proc *** [ 0.045292] *** VALIDATE cgroup *** [ 0.047011] *** VALIDATE cgroup2 *** [ 0.048350] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.049160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.050013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.051030] Spectre V2 : User space: Vulnerable [ 0.052011] Speculative Store Bypass: Vulnerable [ 0.055000] debug: unmapping init [mem 0xffffffff94659000-0xffffffff94660fff] [ 0.057257] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.059077] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.060028] ... version: 2 [ 0.061014] ... bit width: 48 [ 0.062015] ... generic registers: 4 [ 0.063012] ... value mask: 0000ffffffffffff [ 0.064016] ... max period: 00007fffffffffff [ 0.065015] ... fixed-purpose events: 3 [ 0.066013] ... event mask: 000000070000000f [ 0.067424] rcu: Hierarchical SRCU implementation. [ 0.069745] smp: Bringing up secondary CPUs ... [ 0.070756] x86: Booting SMP configuration: [ 0.071055] .... node #0, CPUs: #1 #2 #3 [ 0.084245] smp: Brought up 1 node, 4 CPUs [ 0.086034] smpboot: Max logical packages: 1 [ 0.087013] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.139345] node 0 deferred pages initialised in 49ms [ 0.146282] devtmpfs: initialized [ 0.149007] x86/mm: Memory block size: 128MB [ 0.153576] gcov: version magic: 0x41383552 [ 0.157545] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165395] pinctrl core: initialized pinctrl subsystem [ 0.168019] [ 0.168907] ************************************************************* [ 0.172014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176023] ** ** [ 0.180014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.184015] ** ** [ 0.188017] ** This means that this kernel is built to expose internal ** [ 0.190013] ** IOMMU data structures, which may compromise security on ** [ 0.194016] ** your system. ** [ 0.198070] ** ** [ 0.203012] ** If you see this message and you are not debugging the ** [ 0.208013] ** kernel, report this immediately to your vendor! ** [ 0.211019] ** ** [ 0.213018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.216019] ************************************************************* [ 0.221571] NET: Registered protocol family 16 [ 0.226499] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.230066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.234075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.241132] cpuidle: using governor menu [ 0.244387] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.248712] PCI: Using configuration type 1 for base access [ 0.250136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.267052] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.268021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.272180] cryptd: max_cpu_qlen set to 1000 [ 0.275226] ACPI: Added _OSI(Module Device) [ 0.276161] ACPI: Added _OSI(Processor Device) [ 0.279018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.281013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.287719] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.295595] ACPI: Interpreter enabled [ 0.298180] ACPI: PM: (supports S0 S3 S4 S5) [ 0.301022] ACPI: Using IOAPIC for interrupt routing [ 0.304101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.308432] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.328937] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.332055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.333019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.339092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.348124] acpiphp: Slot [2] registered [ 0.350222] acpiphp: Slot [5] registered [ 0.353209] acpiphp: Slot [6] registered [ 0.358242] acpiphp: Slot [7] registered [ 0.361437] acpiphp: Slot [8] registered [ 0.365159] acpiphp: Slot [9] registered [ 0.368256] acpiphp: Slot [10] registered [ 0.372163] acpiphp: Slot [3] registered [ 0.374124] acpiphp: Slot [4] registered [ 0.375000] acpiphp: Slot [11] registered [ 0.375722] acpiphp: Slot [12] registered [ 0.379134] acpiphp: Slot [13] registered [ 0.382172] acpiphp: Slot [14] registered [ 0.384155] acpiphp: Slot [15] registered [ 0.387152] acpiphp: Slot [16] registered [ 0.389116] acpiphp: Slot [17] registered [ 0.392120] acpiphp: Slot [18] registered [ 0.393110] acpiphp: Slot [19] registered [ 0.394177] acpiphp: Slot [20] registered [ 0.397134] acpiphp: Slot [21] registered [ 0.399150] acpiphp: Slot [22] registered [ 0.402136] acpiphp: Slot [23] registered [ 0.405359] acpiphp: Slot [24] registered [ 0.407153] acpiphp: Slot [25] registered [ 0.410264] acpiphp: Slot [26] registered [ 0.412196] acpiphp: Slot [27] registered [ 0.415153] acpiphp: Slot [28] registered [ 0.417191] acpiphp: Slot [29] registered [ 0.419130] acpiphp: Slot [30] registered [ 0.421135] acpiphp: Slot [31] registered [ 0.424089] PCI host bridge to bus 0000:00 [ 0.425000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.428033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.432024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.435168] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.439025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.440000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.442257] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.446619] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.450787] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.461000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.468102] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.471019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.474022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.478020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.479000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.479000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.482044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.485869] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.491069] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.501000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.505013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.511908] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.523016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.533021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.561022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.575194] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.581022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.594018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.618022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.629874] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.637098] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.643016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.666015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.673927] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.682018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.692020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.711017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.724000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.731162] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.734019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.757502] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.772000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.782018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.793018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.825019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.839209] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.843583] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.848622] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.853396] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.858282] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.867059] iommu: Default domain type: Passthrough [ 0.871970] SCSI subsystem initialized [ 0.872137] ACPI: bus type USB registered [ 0.875283] usbcore: registered new interface driver usbfs [ 0.879997] usbcore: registered new interface driver hub [ 0.884099] usbcore: registered new device driver usb [ 0.889324] pps_core: LinuxPPS API ver. 1 registered [ 0.892016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.897062] PTP clock support registered [ 0.901110] EDAC MC: Ver: 3.0.0 [ 0.905173] PCI: Using ACPI for IRQ routing [ 0.910283] NetLabel: Initializing [ 0.914012] NetLabel: domain hash size = 128 [ 0.916012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.920033] NetLabel: unlabeled traffic allowed by default [ 0.923554] vgaarb: loaded [ 0.927341] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.930017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.940556] clocksource: Switched to clocksource kvm-clock [ 1.315985] VFS: Disk quotas dquot_6.6.0 [ 1.319514] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.325767] *** VALIDATE ramfs *** [ 1.329233] *** VALIDATE hugetlbfs *** [ 1.335529] pnp: PnP ACPI init [ 1.341546] pnp: PnP ACPI: found 6 devices [ 1.387550] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.395198] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.399498] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.406771] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.410255] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.415249] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.421654] NET: Registered protocol family 2 [ 1.425934] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.434923] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.439557] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.445584] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.449598] TCP: Hash tables configured (established 65536 bind 65536) [ 1.455396] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.461833] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.468670] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.474256] NET: Registered protocol family 1 [ 1.483558] RPC: Registered named UNIX socket transport module. [ 1.488981] RPC: Registered udp transport module. [ 1.492577] RPC: Registered tcp transport module. [ 1.495108] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.499765] NET: Registered protocol family 44 [ 1.501621] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.503831] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.505858] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.507920] PCI: CLS 0 bytes, default 64 [ 1.510471] Unpacking initramfs... [ 4.588725] debug: unmapping init [mem 0xffff97077cc54000-0xffff97077ffbffff] [ 4.599379] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.607740] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.617261] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 6.373121] Initialise system trusted keyrings [ 6.375481] Key type blacklist registered [ 6.378616] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.393113] zbud: loaded [ 6.397610] *** VALIDATE nfs *** [ 6.399386] *** VALIDATE nfs4 *** [ 6.402993] pstore: using deflate compression [ 6.420324] Platform Keyring initialized [ 6.668045] NET: Registered protocol family 38 [ 6.672216] Key type asymmetric registered [ 6.675263] Asymmetric key parser 'x509' registered [ 6.679452] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.685660] io scheduler mq-deadline registered [ 6.689593] io scheduler kyber registered [ 6.693326] io scheduler bfq registered [ 6.697216] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.704253] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.710313] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.716987] ACPI: Power Button [PWRF] [ 6.728473] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.741959] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.781676] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.800401] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.831742] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.875238] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.916240] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.923782] Non-volatile memory driver v1.3 [ 6.928713] Linux agpgart interface v0.103 [ 7.000733] virtio_blk virtio1: [vda] 145928 512-byte logical blocks (74.7 MB/71.3 MiB) [ 7.011446] vda: detected capacity change from 0 to 74715136 [ 7.075739] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.084704] vdb: detected capacity change from 0 to 1073741824 [ 7.119647] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.129812] vdc: detected capacity change from 0 to 2621440000 [ 7.198156] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.210051] vdd: detected capacity change from 0 to 2621440000 [ 7.274862] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.284401] vde: detected capacity change from 0 to 4294967296 [ 7.327563] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.332644] vdf: detected capacity change from 0 to 4294967296 [ 7.352490] libphy: Fixed MDIO Bus: probed [ 7.368457] usbcore: registered new interface driver usbserial_generic [ 7.371757] usbserial: USB Serial support registered for generic [ 7.374537] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.379886] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.382190] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.385433] mousedev: PS/2 mouse device common for all mice [ 7.397596] rtc_cmos 00:05: RTC can wake from S4 [ 7.414052] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.414543] rtc_cmos 00:05: registered as rtc0 [ 7.433952] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.439054] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.441918] intel_pstate: CPU model not supported [ 7.455053] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.469282] hid: raw HID events driver (C) Jiri Kosina [ 7.474722] usbcore: registered new interface driver usbhid [ 7.479225] usbhid: USB HID core driver [ 7.482992] drop_monitor: Initializing network drop monitor service [ 7.488485] Initializing XFRM netlink socket [ 7.493140] NET: Registered protocol family 10 [ 7.499783] Segment Routing with IPv6 [ 7.503700] NET: Registered protocol family 17 [ 7.509455] mpls_gso: MPLS GSO support [ 7.518731] RAS: Correctable Errors collector initialized. [ 7.521504] AVX version of gcm_enc/dec engaged. [ 7.522810] AES CTR mode by8 optimization enabled [ 7.625448] sched_clock: Marking stable (7625409682, 0)->(9410112345, -1784702663) [ 7.637762] registered taskstats version 1 [ 7.640759] Loading compiled-in X.509 certificates [ 7.643509] zswap: loaded using pool lzo/zbud [ 7.699330] Key type big_key registered [ 7.721628] Key type encrypted registered [ 7.726378] ima: No TPM chip found, activating TPM-bypass! [ 7.729853] ima: Allocated hash algorithm: sha1 [ 7.734489] ima: No architecture policies found [ 7.737775] evm: Initialising EVM extended attributes: [ 7.740565] evm: security.selinux [ 7.745608] evm: security.ima [ 7.749632] evm: security.capability [ 7.753952] evm: HMAC attrs: 0x1 [ 7.760523] rtc_cmos 00:05: setting system clock to 2026-08-07 11:16:56 UTC (1786101416) [ 7.777566] debug: unmapping init [mem 0xffffffff95603000-0xffffffff957fffff] [ 7.785524] debug: unmapping init [mem 0xffffffff94382000-0xffffffff94658fff] [ 7.797288] Write protecting the kernel read-only data: 28672k [ 7.802582] debug: unmapping init [mem 0xffffffff92a03000-0xffffffff92bfffff] [ 7.807391] debug: unmapping init [mem 0xffffffff93314000-0xffffffff933fffff] [ 7.956530] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.971680] systemd[1]: Detected virtualization kvm. [ 7.975312] systemd[1]: Detected architecture x86-64. [ 7.977437] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.011320] systemd[1]: No hostname configured. [ 8.013197] systemd[1]: Set hostname to . [ 8.015953] random: systemd: uninitialized urandom read (16 bytes read) [ 8.019061] systemd[1]: Initializing machine ID from random generator. [ 8.087991] random: ln: uninitialized urandom read (6 bytes read) [ 8.329378] random: systemd: uninitialized urandom read (16 bytes read) [ 8.336738] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 8.348548] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 8.354561] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.252523] device-mapper: uevent: version 1.0.3 [ 10.255883] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.382045] random: fast init done [ 12.439943] virtio_net virtio0 ens2: renamed from eth0 [ 12.695803] scsi host0: ata_piix [ 12.707337] scsi host1: ata_piix [ 12.709482] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.723817] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.293746] random: crng init done [ 17.295265] random: 7 urandom warning(s) missed due to ratelimiting [ 20.833155] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.360199] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.235842] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.067578] SELinux: Disabled at runtime. [ 26.180366] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.211235] systemd[1]: Detected virtualization kvm. [ 26.219249] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.108606] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.115451] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.134304] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.144352] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.152956] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.168915] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.179441] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ 28.855619] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 29.855724] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.376665] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.560582] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.148149] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.299877] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 38.927158] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit)[ 39.928704] NFS: Registering the id_resolver key type [ 39.937692] Key type id_resolver registered [ 39.943946] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-server login: [ 94.950140] hrtimer: interrupt took 4229747 ns [ 107.368712] libcfs: loading out-of-tree module taints kernel. [ 107.438549] Key type ._llcrypt registered [ 107.441910] Key type .llcrypt registered [ 107.573187] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 129.963558] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 131.839673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 131.865903] alg: No test for adler32 (adler32-zlib) [ 133.418833] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 134.218170] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 136.031189] Key type lgssc registered [ 138.123964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 157.125239] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 203.760508] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 220.424674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 220.477651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.787715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 221.828354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 221.959583] Lustre: lustre-MDT0000: new disk, initializing [ 222.060250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.100071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 228.712808] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.817594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 245.932246] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 245.969150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 245.975579] Lustre: Skipped 1 previous similar message [ 246.066805] Lustre: lustre-MDT0001: new disk, initializing [ 246.160754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.195611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 246.220152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 251.627174] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.031723] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 268.579082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 268.950531] Lustre: lustre-OST0000: new disk, initializing [ 268.954794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 268.964547] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.042978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 272.051421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 272.070918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 272.221259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 278.074357] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 296.016041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 296.178923] Lustre: lustre-OST0001: new disk, initializing [ 296.182898] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 296.190261] Lustre: 9525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 296.339270] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 303.325745] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 304.713514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 304.745500] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 304.869926] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 317.436827] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 328.081464] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 336.133130] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing check_logdir /tmp/testlogs/ [ 343.293186] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing yml_node [ 351.387778] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 355.352725] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 359.663932] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 361.906688] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 7 07:22:48 EDT 2026 [ 386.687719] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 389.612905] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 410.079810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 410.088897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.106559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 412.639812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 412.641804] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 412.647733] Lustre: Skipped 1 previous similar message [ 412.674710] Lustre: Skipped 1 previous similar message [ 417.760429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 417.767598] Lustre: Skipped 5 previous similar messages [ 422.367157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 422.642645] Lustre: server umount lustre-MDT0000 complete [ 427.190391] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101835 with bad export cookie 6003635165172206314 [ 427.192835] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 427.206155] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 428.000024] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.004120] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.018511] Lustre: Skipped 1 previous similar message [ 428.019825] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 428.049282] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 433.126224] LustreError: 12160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 433.126768] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 433.147596] LustreError: 12160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 433.172446] Lustre: Skipped 2 previous similar messages [ 433.752262] Lustre: server umount lustre-MDT0001 complete [ 439.195137] Lustre: server umount lustre-OST0000 complete [ 442.876884] Lustre: server umount lustre-OST0001 complete [ 451.127469] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 463.563925] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 509.471264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.756674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 509.790726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 509.913245] Lustre: lustre-MDT0000: new disk, initializing [ 510.069685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.130575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 516.284878] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 529.313335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.529351] Lustre: 15088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 529.679720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 529.688968] Lustre: Skipped 1 previous similar message [ 529.831445] Lustre: lustre-MDT0001: new disk, initializing [ 530.021652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 530.063823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 530.083146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 535.798978] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 548.817395] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 556.511897] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 567.360552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 567.755675] Lustre: lustre-OST0000: new disk, initializing [ 567.761657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 567.781871] Lustre: 17058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 567.903476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 569.223651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 569.244215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 569.536202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 576.761581] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 588.863777] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 594.289407] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 594.493946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 600.373897] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 600.850861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 605.155684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 605.161070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.178218] Lustre: Skipped 1 previous similar message [ 605.188420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 609.860715] Lustre: server umount lustre-OST0000 complete [ 615.908935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.925787] Lustre: Skipped 2 previous similar messages [ 620.617937] Lustre: server umount lustre-MDT0000 complete [ 621.028611] LustreError: 15100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.042648] LustreError: 15100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 624.926364] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102033 with bad export cookie 6003635165172207847 [ 624.929264] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.943421] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 625.323278] Lustre: server umount lustre-MDT0001 complete [ 633.597075] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 07:27:19 (1786102039) [ 656.935916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.712412] LustreError: 19937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.874865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 663.010189] LustreError: 19938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.890743] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 673.269205] LustreError: 19937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 673.303127] LustreError: 19937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 675.597115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 682.143917] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 692.036825] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 698.510377] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 701.612468] Lustre: Failing over lustre-MDT0000 [ 701.923342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 701.949700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.986049] LustreError: 20666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.068278] Lustre: server umount lustre-MDT0000 complete [ 721.384084] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786102114/real 1786102114] req@ffff9707f900c700 x1872863210686464/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786102130 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 721.410546] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.268697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.950171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 731.975370] Lustre: Skipped 1 previous similar message [ 732.035996] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 737.252501] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 737.263421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 737.286131] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 737.901754] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 748.813292] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 755.685837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.690262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 755.699519] Lustre: Skipped 2 previous similar messages [ 755.708484] Lustre: Skipped 4 previous similar messages [ 760.188208] Lustre: server umount lustre-MDT0000 complete [ 760.803093] LustreError: 19938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 760.829946] LustreError: 19938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 764.490972] LustreError: 19919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102173 with bad export cookie 6003635165172209261 [ 764.492409] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 764.502823] LustreError: 19919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 764.994637] Lustre: server umount lustre-MDT0001 complete [ 775.723638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.200447] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 791.584902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.504619] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 808.239842] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 817.546539] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 820.364879] Lustre: Failing over lustre-MDT0000 [ 820.582979] Lustre: server umount lustre-MDT0000 complete [ 822.757803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 822.766273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 827.879317] LustreError: 23281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 827.906542] LustreError: 23281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 839.140945] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786102231/real 1786102231] req@ffff9706c3c9fb80 x1872863210718080/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786102247 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 839.195462] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 842.735256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.357843] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5351325fc5221a17 [ 848.398277] Lustre: MGC192.168.204.142@tcp: Connection restored to 0@lo (at 0@lo) [ 848.410422] Lustre: Skipped 1 previous similar message [ 848.911799] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 853.987236] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 853.991723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 854.020934] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 854.428228] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 867.519418] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 873.452635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 873.468679] Lustre: Skipped 1 previous similar message [ 879.576220] Lustre: server umount lustre-MDT0000 complete [ 885.325705] LustreError: 23261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102294 with bad export cookie 6003635165172210199 [ 885.332718] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 885.356273] LustreError: 23261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 885.808914] Lustre: server umount lustre-MDT0001 complete [ 899.629321] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 07:31:45 (1786102305) [ 921.984875] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 936.552898] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 990.692902] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1005.452837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.653231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1005.682196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1005.745107] Lustre: lustre-MDT0000: new disk, initializing [ 1005.816102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.823782] Lustre: Skipped 3 previous similar messages [ 1005.836553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1011.676752] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1025.757336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.029996] Lustre: 29665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1026.486153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1032.955689] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1045.086412] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1053.131462] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1062.858501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1063.289540] Lustre: 31636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1065.437565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1065.499853] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1065.578923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1065.611776] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1071.974788] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1084.659286] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1094.095716] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1121.767834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.771361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1121.779090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.779099] Lustre: Skipped 2 previous similar messages [ 1121.799702] Lustre: Skipped 4 previous similar messages [ 1121.835149] LustreError: Skipped 1 previous similar message [ 1125.340574] Lustre: server umount lustre-OST0000 complete [ 1145.311138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1145.533146] Lustre: server umount lustre-MDT0000 complete [ 1147.370145] LustreError: 30537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.405677] LustreError: 30537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 1150.294606] LustreError: 29656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102559 with bad export cookie 6003635165172210612 [ 1150.299035] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.305756] LustreError: 29656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1150.829239] Lustre: server umount lustre-MDT0001 complete [ 1165.185668] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1169.119769] Key type lgssc unregistered [ 1169.489703] LNet: 33621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.495460] LNetError: 33621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.522672] LNet: Removed LNI 192.168.204.142@tcp [ 1170.587293] Key type .llcrypt unregistered [ 1170.591394] Key type ._llcrypt unregistered [ 1201.649656] Key type ._llcrypt registered [ 1201.656453] Key type .llcrypt registered [ 1201.830722] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1203.325355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1203.491974] alg: No test for adler32 (adler32-zlib) [ 1205.260891] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 1205.735315] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1207.551194] Key type lgssc registered [ 1208.858247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1220.654600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1220.686105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.598133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.262243] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1232.869804] LustreError: 35043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.895077] LustreError: 35043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1237.993286] LustreError: 35042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.736059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.279237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1249.031538] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1260.151043] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1268.750869] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1277.391771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1277.808909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.789641] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1287.034900] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1293.211305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1295.370325] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1314.939646] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 07:38:40 (1786102720) [ 1317.454586] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1319.404652] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1330.183522] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 07:38:56 (1786102736) [ 1343.298439] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 07:39:09 (1786102749) [ 1344.458493] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1356.690665] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1356.709316] Lustre: Skipped 2 previous similar messages [ 1358.990734] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 07:39:25 (1786102765) [ 1361.170143] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1361.179682] Lustre: Skipped 2 previous similar messages [ 1371.032739] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1371.046357] Lustre: Skipped 4 previous similar messages [ 1387.256539] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1387.263549] Lustre: Skipped 5 previous similar messages [ 1389.645697] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 07:39:55 (1786102795) [ 1393.262176] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1398.135921] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1410.667366] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 07:40:16 (1786102816) [ 1428.050283] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1428.057928] Lustre: Skipped 4 previous similar messages [ 1430.921427] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 07:40:36 (1786102836) [ 1450.981280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1450.996399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.020766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.008608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.011880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1452.041508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.924794] Lustre: server umount lustre-OST0000 complete [ 1462.246035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1462.251987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.260505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.260513] Lustre: Skipped 1 previous similar message [ 1462.294957] Lustre: Skipped 1 previous similar message [ 1465.421339] Lustre: server umount lustre-MDT0000 complete [ 1467.368749] LustreError: 37833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.407607] LustreError: 37833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1469.945726] LustreError: 35023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102878 with bad export cookie 14519745251379363794 [ 1469.950696] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1469.952652] LustreError: 35023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1470.321133] Lustre: server umount lustre-MDT0001 complete [ 1482.938258] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1486.209920] Key type lgssc unregistered [ 1486.575300] LNet: 42912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.586033] LNetError: 42912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.604233] LNet: Removed LNI 192.168.204.142@tcp [ 1487.627282] Key type .llcrypt unregistered [ 1487.629355] Key type ._llcrypt unregistered [ 1509.989546] Key type ._llcrypt registered [ 1509.992507] Key type .llcrypt registered [ 1510.098269] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 1531.327461] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1532.791018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1532.831885] alg: No test for adler32 (adler32-zlib) [ 1533.887273] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 1534.082122] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1535.792373] Key type lgssc registered [ 1537.759598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1583.790410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1583.864730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.135036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1585.180909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1585.316772] Lustre: lustre-MDT0000: new disk, initializing [ 1585.453094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1585.493945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1592.626881] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1604.723037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.973481] Lustre: 46075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1605.113813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1605.129424] Lustre: Skipped 1 previous similar message [ 1605.275522] Lustre: lustre-MDT0001: new disk, initializing [ 1605.503874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1605.604975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1605.639433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1611.604899] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1625.208135] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1635.355275] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1643.884762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1644.246621] Lustre: lustre-OST0000: new disk, initializing [ 1644.250918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1644.261253] Lustre: 48046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1644.393733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1650.788572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1650.809693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1650.955672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1652.376551] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1664.818575] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1670.912941] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1671.179660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1676.487790] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1676.742708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1681.890307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.894886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1681.917795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.921718] Lustre: Skipped 1 previous similar message [ 1681.963350] LustreError: Skipped 1 previous similar message [ 1685.314844] Lustre: server umount lustre-OST0000 complete [ 1692.133572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.160398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1692.172660] Lustre: Skipped 2 previous similar messages [ 1696.644295] Lustre: server umount lustre-MDT0000 complete [ 1701.639715] LustreError: 46067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786103110 with bad export cookie 12435777494311481415 [ 1701.648615] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1701.655626] LustreError: 46067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1702.104647] Lustre: server umount lustre-MDT0001 complete [ 1715.746842] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:45:22 (1786103122) [ 1723.546882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.113074] LustreError: 50506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1724.130462] LustreError: 50506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1724.203060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.519364] LustreError: 50507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1729.617515] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1733.605839] LustreError: 50506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1738.724526] LustreError: 50507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1739.985990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1740.281151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1746.300274] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1756.548696] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1764.952241] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1772.879124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1773.134867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.970719] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1791.643873] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1802.170844] LustreError: 53041:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1803.300430] Lustre: Modifying parameter general.timeout=1 in log params [ 1804.475839] Lustre: Modifying parameter general.timeout=2 in log params [ 1807.326757] Lustre: Modifying parameter general.timeout=5 in log params [ 1807.330653] Lustre: Skipped 2 previous similar messages [ 1811.550106] Lustre: Modifying parameter general.timeout=9 in log params [ 1811.555810] Lustre: Skipped 3 previous similar messages [ 1820.394981] Lustre: Modifying parameter general.timeout=16 in log params [ 1820.398740] Lustre: Skipped 6 previous similar messages [ 1836.907482] Lustre: Modifying parameter general.timeout=31 in log params [ 1836.915963] Lustre: Skipped 14 previous similar messages [ 1869.573255] Lustre: Modifying parameter general.timeout=61 in log params [ 1869.579624] Lustre: Skipped 29 previous similar messages [ 1934.463854] Lustre: Modifying parameter general.timeout=117 in log params [ 1934.467771] Lustre: Skipped 55 previous similar messages [ 1964.910971] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 07:49:31 (1786103371) [ 1969.743914] Lustre: 59998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1972.291809] Lustre: 60096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1974.750471] Lustre: 60195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1979.634372] Lustre: 60408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1979.651924] Lustre: 60408:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2016.587927] Lustre: 61880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 2016.604308] Lustre: 61880:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2016.640200] LustreError: 61880:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 2022.429027] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:50:28 (1786103428) [ 2041.321267] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.329702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2041.353195] Lustre: Skipped 2 previous similar messages [ 2041.384536] Lustre: Skipped 2 previous similar messages [ 2046.228490] Lustre: server umount lustre-MDT0000 complete [ 2046.439152] LustreError: 50507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.458059] LustreError: 50507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2050.517488] LustreError: 50487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786103459 with bad export cookie 12435777494311482290 [ 2050.523574] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2050.542791] LustreError: 50487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2050.819785] Lustre: server umount lustre-MDT0001 complete [ 2065.373672] Lustre: server umount lustre-OST0000 complete [ 2096.949346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.288480] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2097.333819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2097.340144] Lustre: Skipped 35 previous similar messages [ 2097.635276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.197181] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2111.405470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.526882] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2111.569769] Lustre: 64411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2117.520400] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2125.700239] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2131.925531] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2139.023699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2139.168941] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2139.405663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.412032] Lustre: Skipped 1 previous similar message [ 2145.714905] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2155.184605] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2157.399348] Lustre: 66982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2169.313688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2169.320587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.332604] Lustre: Skipped 1 previous similar message [ 2169.343193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.284922] Lustre: server umount lustre-OST0000 complete [ 2178.530113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2178.538264] LustreError: Skipped 1 previous similar message [ 2178.542076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.553034] Lustre: Skipped 1 previous similar message [ 2178.558346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.566737] Lustre: Skipped 1 previous similar message [ 2181.437094] Lustre: server umount lustre-MDT0000 complete [ 2185.184609] LustreError: 65169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.220386] LustreError: 65169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2185.259871] LustreError: 64405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786103593 with bad export cookie 12435777494311485552 [ 2185.260545] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2185.295574] LustreError: 64405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2185.605382] Lustre: server umount lustre-MDT0001 complete [ 2198.462459] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2201.506882] Key type lgssc unregistered [ 2201.867250] LNet: 68199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2201.874338] LNetError: 68199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.919645] LNet: Removed LNI 192.168.204.142@tcp [ 2203.721710] Key type .llcrypt unregistered [ 2203.723599] Key type ._llcrypt unregistered [ 2232.484612] Key type ._llcrypt registered [ 2232.487491] Key type .llcrypt registered [ 2232.592873] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2233.637672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2233.653031] alg: No test for adler32 (adler32-zlib) [ 2234.660474] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 2234.840566] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2236.495182] Key type lgssc registered [ 2237.608930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2245.865960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2245.903517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2247.454737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.436689] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2257.891739] LustreError: 69613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.924606] LustreError: 69613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2262.401617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.997610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2268.346410] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2276.593156] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2283.901878] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2291.889871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2292.488283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2300.258304] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2309.228721] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2318.041706] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2329.573194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.573626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2329.578388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2329.592943] Lustre: Skipped 1 previous similar message [ 2333.406535] Lustre: server umount lustre-MDT0000 complete [ 2334.693920] LustreError: 69614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.719868] LustreError: 69614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2337.763288] LustreError: 72652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786103746 with bad export cookie 3159681829073012701 [ 2337.767146] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2337.775447] LustreError: 72652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2338.127843] Lustre: server umount lustre-MDT0001 complete [ 2352.365459] Lustre: server umount lustre-OST0000 complete [ 2380.465077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.746657] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2380.772675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2381.044119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.066964] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2394.575611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.639981] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2394.658390] Lustre: 74481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2395.072659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2399.641449] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2407.510682] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2413.731508] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2421.073223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2421.255665] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2421.543912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.584146] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2435.368798] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2437.181565] Lustre: 77057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2439.543393] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2444.257378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2444.266294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.278297] Lustre: Skipped 1 previous similar message [ 2444.284756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.290480] Lustre: Skipped 2 previous similar messages [ 2449.897885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2449.917315] Lustre: Skipped 2 previous similar messages [ 2450.390545] Lustre: server umount lustre-OST0000 complete [ 2455.012766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.021308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2455.032223] Lustre: Skipped 2 previous similar messages [ 2455.041571] Lustre: Skipped 1 previous similar message [ 2460.133541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2460.150119] Lustre: Skipped 1 previous similar message [ 2460.477685] Lustre: server umount lustre-MDT0000 complete [ 2464.439204] LustreError: 74473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786103873 with bad export cookie 3159681829073013534 [ 2464.455150] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2464.456198] LustreError: 74473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2464.834470] Lustre: server umount lustre-MDT0001 complete [ 2475.281490] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2478.218413] Key type lgssc unregistered [ 2478.490847] LNet: 78277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2478.502895] LNetError: 78277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2478.521598] LNet: Removed LNI 192.168.204.142@tcp [ 2479.430780] Key type .llcrypt unregistered [ 2479.432755] Key type ._llcrypt unregistered [ 2507.517551] Key type ._llcrypt registered [ 2507.527032] Key type .llcrypt registered [ 2507.715650] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2509.457555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2509.479826] alg: No test for adler32 (adler32-zlib) [ 2510.559304] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 2510.770217] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2512.463334] Key type lgssc registered [ 2513.898211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2523.032181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2523.072593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.472399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.738139] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2534.886521] LustreError: 79692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.905935] LustreError: 79692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2537.898919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.376838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2543.534245] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2552.648431] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2559.677208] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2567.666598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2568.224661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.823416] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2589.069564] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2598.773808] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2600.041393] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2601.152365] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2603.445512] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2603.461384] Lustre: Skipped 1 previous similar message [ 2607.644641] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2607.648663] Lustre: Skipped 3 previous similar messages [ 2616.090330] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2616.099229] Lustre: Skipped 6 previous similar messages [ 2632.265620] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2632.272553] Lustre: Skipped 12 previous similar messages [ 2665.044944] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2665.053357] Lustre: Skipped 25 previous similar messages [ 2726.021388] Lustre: 87824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2729.167151] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:02:15 (1786104135) [ 2731.799707] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2734.486221] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:02:20 (1786104140) [ 2741.223368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.233070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2741.241827] Lustre: Skipped 1 previous similar message [ 2741.242472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2741.286584] Lustre: Skipped 1 previous similar message [ 2745.526492] Lustre: server umount lustre-OST0000 complete [ 2751.458893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.461884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.485713] Lustre: Skipped 1 previous similar message [ 2755.692646] Lustre: server umount lustre-MDT0000 complete [ 2756.576238] LustreError: 79687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.589831] LustreError: 79687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2759.522106] LustreError: 79674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104168 with bad export cookie 17950584220703561018 [ 2759.528570] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2759.531235] LustreError: 79674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2759.858573] Lustre: server umount lustre-MDT0001 complete [ 2773.022149] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2775.626711] Key type lgssc unregistered [ 2775.966421] LNet: 89075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2775.974299] LNetError: 89075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2777.001090] LNet: Removed LNI 192.168.204.142@tcp [ 2777.904224] Key type .llcrypt unregistered [ 2777.906370] Key type ._llcrypt unregistered [ 2782.659958] Key type ._llcrypt registered [ 2782.662897] Key type .llcrypt registered [ 2782.814494] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2789.343749] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules [ 2790.965937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2791.153933] alg: No test for adler32 (adler32-zlib) [ 2792.546473] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 2792.885332] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2794.535624] Key type lgssc registered [ 2796.780959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2798.529689] LustreError: 90331:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2821.124466] LustreError: 90331:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2822.232928] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2834.473138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2834.494795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.070192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.940178] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2845.674822] LustreError: 91354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.694549] LustreError: 91354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2850.785800] LustreError: 91353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.871243] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:04:20 (1786104260) [ 2855.912663] LustreError: 91354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.259634] LustreError: 91385:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2860.283642] LustreError: 91385:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2860.305380] LustreError: 91385:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 25, retries 0, failed: rc = -5 [ 2860.878981] Lustre: server umount lustre-MDT0000 complete [ 2880.059326] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2883.950983] Key type lgssc unregistered [ 2884.333510] LNet: 92885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2884.349658] LNetError: 92885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2885.416062] LNet: Removed LNI 192.168.204.142@tcp [ 2886.690384] Key type .llcrypt unregistered [ 2886.694782] Key type ._llcrypt unregistered [ 2912.394583] Key type ._llcrypt registered [ 2912.397611] Key type .llcrypt registered [ 2912.641630] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2914.386259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2914.593042] alg: No test for adler32 (adler32-zlib) [ 2915.644401] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 2915.974745] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2917.823159] Key type lgssc registered [ 2919.439501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2930.560286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2930.582411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.154396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.273211] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2941.445756] LustreError: 94105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.484644] LustreError: 94105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2946.533878] LustreError: 94106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.075568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2951.683727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2957.746722] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2968.029892] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2977.053644] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2984.759124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.276037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.629409] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3003.725623] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3055.601816] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:07:41 (1786104461) [ 3058.089881] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 3060.727701] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:07:46 (1786104466) [ 3069.408824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.423029] Lustre: Skipped 1 previous similar message [ 3069.424187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3069.441235] Lustre: Skipped 2 previous similar messages [ 3071.096946] Lustre: server umount lustre-MDT0000 complete [ 3074.529690] LustreError: 94106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.907889] LustreError: 94086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104484 with bad export cookie 2698569783407990839 [ 3075.923302] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3075.929669] LustreError: 94086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3076.274914] Lustre: server umount lustre-MDT0001 complete [ 3090.705798] Lustre: server umount lustre-OST0000 complete [ 3104.587909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.902521] LustreError: 98382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.932080] LustreError: 98382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3105.056304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.370525] LustreError: 98383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.431619] LustreError: 98383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3112.063376] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3119.590104] LustreError: 98383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.614807] LustreError: 98383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3122.443150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.892460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3128.325139] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3138.298694] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3148.252675] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3165.378714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3165.627370] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3165.638856] LustreError: 98370:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3165.673233] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3165.682527] LustreError: 100467:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3165.697550] LustreError: 100467:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3165.708916] LustreError: 100467:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3165.792380] Lustre: server umount lustre-OST0000 complete [ 3165.798499] LustreError: 100467:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3177.918370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.089841] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3178.096799] LustreError: 98373:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3178.122170] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3178.142858] LustreError: 100934:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3178.154773] LustreError: 100934:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3178.166518] LustreError: 100934:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3178.229252] Lustre: server umount lustre-OST0000 complete [ 3178.232472] LustreError: 100934:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3189.847743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3190.053429] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3190.058102] Lustre: Found index 0 for lustre-OST0000, updating log [ 3190.063872] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3190.154576] Lustre: lustre-OST0000: new disk, initializing [ 3190.159404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3190.166823] Lustre: 101469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3190.558115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3199.528736] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3203.854589] Lustre: Failing over lustre-OST0000 [ 3204.065653] Lustre: server umount lustre-OST0000 complete [ 3209.699353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3209.713519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3209.732409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.331233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.344699] Lustre: Skipped 3 previous similar messages [ 3218.481860] Lustre: server umount lustre-MDT0000 complete [ 3220.319143] Lustre: 93416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786104613/real 1786104613] req@00000000144b2411 x1872866127750784/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786104629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3220.344960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.375606] Lustre: Skipped 1 previous similar message [ 3220.402267] LustreError: 98377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3220.462891] LustreError: 98377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3223.045554] LustreError: 100951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104631 with bad export cookie 2698569783407993590 [ 3223.046289] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3223.061593] LustreError: 100951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3223.348532] Lustre: server umount lustre-MDT0001 complete [ 3233.688620] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:10:40 (1786104640) [ 3252.811451] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3265.460155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.966193] LustreError: 103818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3266.096984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3271.775334] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3281.424123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.199822] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3291.760942] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3299.795842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3300.631659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.646875] Lustre: Skipped 1 previous similar message [ 3300.664938] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3306.790399] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3309.379670] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3312.009298] Lustre: lustre-OST0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 3312.014584] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3312.184986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3320.929476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3321.025749] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3321.231841] Lustre: lustre-OST0001: new disk, initializing [ 3321.243502] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3321.262947] Lustre: 106140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3327.630908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3327.641871] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3327.815548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3330.224155] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3338.760927] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3351.657532] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3358.700745] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.711516] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3358.725627] Lustre: Skipped 1 previous similar message [ 3358.762137] Lustre: Skipped 2 previous similar messages [ 3362.861518] Lustre: server umount lustre-MDT0001 complete [ 3363.827252] LustreError: 103819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3363.856521] LustreError: 103819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3375.519064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.612937] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3375.617435] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3375.664817] Lustre: 103808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3376.624840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3376.639913] Lustre: Skipped 1 previous similar message [ 3376.766772] LustreError: 107658:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3376.782723] LustreError: 107658:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3376.798364] LustreError: 103810:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3376.969846] LustreError: 107668:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.142@tcp: cfg command failed: rc = -17 [ 3376.998554] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3376.998554] [ 3381.743950] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3381.748635] Lustre: Skipped 1 previous similar message [ 3383.049583] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3394.075981] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:13:20 (1786104800) [ 3419.410404] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3444.721523] Lustre: 109778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3474.265720] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3479.175811] Lustre: 110914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3570.659530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3570.679531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3570.700368] Lustre: Skipped 1 previous similar message [ 3570.717689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3570.729599] Lustre: Skipped 1 previous similar message [ 3574.198101] Lustre: server umount lustre-MDT0000 complete [ 3576.290431] LustreError: 104560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.313789] LustreError: 104560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 3578.737847] LustreError: 103800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104987 with bad export cookie 2698569783407994598 [ 3578.750568] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3578.751473] LustreError: 103800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3579.278694] Lustre: server umount lustre-MDT0001 complete [ 3594.736086] Lustre: server umount lustre-OST0000 complete [ 3597.800457] Lustre: 93417:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786104990/real 1786104990] req@ffff9706cec33100 x1872866128026880/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786105006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3597.847947] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.870128] Lustre: Skipped 3 previous similar messages [ 3599.419838] Lustre: server umount lustre-OST0001 complete [ 3606.638259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3619.092366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3629.117129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3642.405502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3666.902973] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3680.905255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.921218] Lustre: lustre-MDT0000: reset Object Index mappings [ 3681.456878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.632208] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3697.314780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.327711] Lustre: lustre-MDT0001: reset Object Index mappings [ 3702.975717] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3706.613558] Lustre: 116012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3714.494361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3722.504237] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 3722.795449] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3726.633722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3732.222654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3738.147050] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3738.163703] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3741.416303] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3749.185522] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3753.852155] Lustre: 117888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3810.699910] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:20:16 (1786105216) [ 3835.873684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3835.883405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.904314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3835.918370] Lustre: Skipped 4 previous similar messages [ 3841.131312] Lustre: server umount lustre-MDT0000 complete [ 3844.581191] LustreError: 114861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3844.624391] LustreError: 114861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 3846.155685] LustreError: 114847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786105254 with bad export cookie 2698569783408041673 [ 3846.159896] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3846.174039] LustreError: 114847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3852.638884] Lustre: server umount lustre-MDT0001 complete [ 3862.685214] Lustre: server umount lustre-OST0000 complete [ 3872.813733] Lustre: server umount lustre-OST0001 complete [ 3881.453239] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3895.620508] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3938.386156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.783668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3938.830307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3938.968810] Lustre: lustre-MDT0000: new disk, initializing [ 3939.063917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3939.071772] Lustre: Skipped 3 previous similar messages [ 3939.103578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3945.033131] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3953.475805] Lustre: server umount lustre-MDT0000 complete [ 3964.427491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.699748] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3964.710701] Lustre: 122165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3964.722637] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3971.332767] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3977.635157] Lustre: server umount lustre-MDT0000 complete [ 3990.613092] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:23:16 (1786105396) [ 3993.024743] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3995.336984] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:23:22 (1786105402) [ 4017.487840] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4030.202539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.398104] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4046.027415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.144735] Lustre: 124233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4046.199694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4046.203401] Lustre: Skipped 1 previous similar message [ 4046.271337] Lustre: lustre-MDT0001: new disk, initializing [ 4046.474147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4046.521065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4052.677257] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4058.636961] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4066.916458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4067.488521] Lustre: lustre-OST0000: new disk, initializing [ 4067.495497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4067.509791] Lustre: 125813:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4068.715930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4068.728083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4068.864472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4075.497889] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4088.734099] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4088.956539] Lustre: lustre-OST0001: new disk, initializing [ 4088.967950] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4088.984830] Lustre: 126685:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4090.936952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4090.958394] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4091.198621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4096.614583] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4106.954616] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4111.119831] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4535.161307] Lustre: 133114:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4535.168863] Lustre: 133114:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4535.172396] Lustre: 133114:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4535.177195] Lustre: 133114:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4535.181014] Lustre: 133114:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4535.185146] Lustre: 133114:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4838.553719] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:37:24 (1786106244) [ 4855.264473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4855.269373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4855.276835] Lustre: Skipped 5 previous similar messages [ 4855.282899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4855.287353] Lustre: Skipped 8 previous similar messages [ 4857.318803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4857.335983] Lustre: Skipped 3 previous similar messages [ 4861.089225] Lustre: server umount lustre-MDT0000 complete [ 4862.438833] LustreError: 125808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.457966] LustreError: 125808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4864.976814] LustreError: 124226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786106273 with bad export cookie 2698569783408110231 [ 4864.984501] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.992865] LustreError: 124226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4867.556393] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4867.564895] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4867.567833] Lustre: Skipped 4 previous similar messages [ 4867.578518] Lustre: Skipped 1 previous similar message [ 4879.327271] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4879.843950] Lustre: server umount lustre-MDT0001 complete [ 4890.496473] Lustre: server umount lustre-OST0000 complete [ 4901.332787] Lustre: server umount lustre-OST0001 complete [ 4910.657885] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 4924.380930] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4968.598735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4968.992916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4969.049775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4969.124433] Lustre: lustre-MDT0000: new disk, initializing [ 4969.199643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4969.206616] Lustre: Skipped 5 previous similar messages [ 4969.223126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4975.071778] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4988.220283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4988.489672] Lustre: 136178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4988.593094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4988.602791] Lustre: Skipped 1 previous similar message [ 4988.724892] Lustre: lustre-MDT0001: new disk, initializing [ 4988.918958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4988.927380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4994.254663] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5005.834308] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5013.335833] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5021.049287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5021.312954] Lustre: lustre-OST0000: new disk, initializing [ 5021.316564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5021.322542] Lustre: 138147:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5023.010969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5023.030123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5023.174458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5028.786512] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5039.543102] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5053.516618] Lustre: lustre-MDD0000: changelog on [ 5056.733453] Lustre: lustre-MDD0001: changelog on [ 6956.957503] Lustre: lustre-MDD0001: changelog off [ 6959.445299] Lustre: 136185:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6960.568746] Lustre: lustre-MDD0000: changelog off [ 6967.783304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6967.796846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6967.812711] Lustre: Skipped 1 previous similar message [ 6967.823225] Lustre: Skipped 4 previous similar messages [ 6972.904427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6972.908780] Lustre: Skipped 2 previous similar messages [ 6973.438527] Lustre: server umount lustre-OST0000 complete [ 6978.019356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6978.022159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6978.043158] Lustre: Skipped 1 previous similar message [ 6983.137033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6983.142736] Lustre: Skipped 2 previous similar messages [ 6984.041343] Lustre: server umount lustre-MDT0000 complete [ 6988.260367] LustreError: 136189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.304596] LustreError: 136189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6988.922678] LustreError: 136169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786108397 with bad export cookie 2698569783408938289 [ 6988.941382] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6988.956629] LustreError: 136169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6989.480506] Lustre: server umount lustre-MDT0001 complete [ 7004.495596] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7008.891314] Key type lgssc unregistered [ 7009.385404] LNet: 147310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7009.401735] LNetError: 147310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7009.426864] LNet: Removed LNI 192.168.204.142@tcp [ 7010.744361] Key type .llcrypt unregistered [ 7010.747404] Key type ._llcrypt unregistered [ 7037.543674] Key type ._llcrypt registered [ 7037.547536] Key type .llcrypt registered [ 7037.762612] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7057.852782] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7059.668958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7059.783561] alg: No test for adler32 (adler32-zlib) [ 7061.040482] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 7061.546595] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 7063.383131] Key type lgssc registered [ 7065.304723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7108.381223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7108.432067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.717736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7109.743776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7109.816297] Lustre: lustre-MDT0000: new disk, initializing [ 7109.906809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7109.927795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7116.027418] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7128.767667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7128.973573] Lustre: 150514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7129.085649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7129.092265] Lustre: Skipped 1 previous similar message [ 7129.226773] Lustre: lustre-MDT0001: new disk, initializing [ 7129.404374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7129.475390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7129.502186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7135.789667] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7149.051867] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7157.471626] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7166.469744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7166.771717] Lustre: lustre-OST0000: new disk, initializing [ 7166.775797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7166.782367] Lustre: 152485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7166.913498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7176.365726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7176.387127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7176.569859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7176.857736] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7190.270657] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7197.509217] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_module obdecho/obdecho [ 7199.937372] LustreError: 153489:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7199.953893] LustreError: 153489:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7202.448618] LustreError: 153585:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7216.610834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7216.616288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7216.634925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7217.636516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7217.648482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7217.658174] Lustre: Skipped 1 previous similar message [ 7217.666457] Lustre: Skipped 2 previous similar messages [ 7222.720720] Lustre: server umount lustre-MDT0000 complete [ 7227.330407] LustreError: 150506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786108636 with bad export cookie 7211968194491832761 [ 7227.345426] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7227.349586] LustreError: 150506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7227.673774] Lustre: server umount lustre-MDT0001 complete [ 7242.916857] Lustre: server umount lustre-OST0000 complete [ 7268.462730] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7279.936786] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7329.728560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7329.990427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7330.018519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7330.083609] Lustre: lustre-MDT0000: new disk, initializing [ 7330.242617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7330.289329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7335.623422] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7349.275180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7349.561806] Lustre: 157306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7349.641225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7349.647765] Lustre: Skipped 1 previous similar message [ 7349.725875] Lustre: lustre-MDT0001: new disk, initializing [ 7349.838050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7349.872914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7349.881994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7355.015373] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7369.109161] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7376.642383] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7386.811242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7387.256844] Lustre: lustre-OST0000: new disk, initializing [ 7387.269671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7387.281856] Lustre: 159274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7387.509281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7393.431516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7393.450910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7393.802974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7396.744615] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7408.082250] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7414.579152] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7414.803195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7421.618488] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7421.992598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7424.482046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7424.488252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7424.503636] LustreError: Skipped 1 previous similar message [ 7424.504592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7424.529407] Lustre: Skipped 1 previous similar message [ 7429.600973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7429.623788] Lustre: Skipped 2 previous similar messages [ 7430.765859] Lustre: server umount lustre-OST0000 complete [ 7436.771184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7436.782101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7436.813611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7442.461679] Lustre: server umount lustre-MDT0000 complete [ 7444.962182] LustreError: 158116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7444.979984] LustreError: 158116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7446.782626] LustreError: 157299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786108855 with bad export cookie 7211968194491833720 [ 7446.784207] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7446.791126] LustreError: 157299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7447.140332] Lustre: server umount lustre-MDT0001 complete [ 7451.334423] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:20:57 (1786108857) [ 7453.502563] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7456.724749] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:21:02 (1786108862) [ 7467.619546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7468.621940] LustreError: 161686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7468.834294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7474.146151] LustreError: 161687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7475.236394] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7479.266589] LustreError: 161686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7484.392577] LustreError: 161687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7484.910259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7491.260409] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7501.898201] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7509.627717] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7519.052216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7519.348971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7519.357973] Lustre: Skipped 1 previous similar message [ 7527.461169] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7536.936855] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7549.077590] Lustre: server umount lustre-MDT0000 complete [ 7550.445290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.458337] LustreError: 161682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.483462] Lustre: Skipped 3 previous similar messages [ 7550.584578] LustreError: 161682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7554.277862] LustreError: 161668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786108963 with bad export cookie 7211968194491834602 [ 7554.279439] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7554.305939] LustreError: 161668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7555.555417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7555.558095] Lustre: Skipped 2 previous similar messages [ 7560.959572] Lustre: server umount lustre-MDT0001 complete [ 7569.457703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7585.385860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7585.569443] LustreError: 165532:0:(llog.c:1655:llog_backup()) MGC192.168.204.142@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7585.583707] Lustre: 165532:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7589.345676] LustreError: 165537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7589.366877] LustreError: 165537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7624.482995] LustreError: 165538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7624.498621] LustreError: 165538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [ 7646.175969] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7646.187914] Lustre: 165532:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7687.181244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7688.998260] LustreError: 165537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7689.027996] LustreError: 165537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 7692.271059] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7692.802691] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7702.000830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7709.674598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7725.026840] LustreError: 148733:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000aecee870 x1872870475183616/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7729.370649] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7730.667297] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7738.454457] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7740.689641] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7747.710558] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7749.791917] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7766.497591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7766.507245] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7766.511060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7766.524309] Lustre: Skipped 2 previous similar messages [ 7766.539573] Lustre: Skipped 4 previous similar messages [ 7767.677294] Lustre: server umount lustre-MDT0000 complete [ 7770.966025] LustreError: 163534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786109179 with bad export cookie 7211968194491841091 [ 7770.968137] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7770.977607] LustreError: 163534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7771.384784] Lustre: server umount lustre-MDT0001 complete [ 7785.856888] Lustre: server umount lustre-OST0000 complete [ 7805.605629] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7815.377989] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7844.367124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7844.614669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7844.649276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7844.706501] Lustre: lustre-MDT0000: new disk, initializing [ 7844.767283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7844.773509] Lustre: Skipped 1 previous similar message [ 7844.786702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7848.335921] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7860.383458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7860.593641] Lustre: 170641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7860.874404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7867.216497] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7880.879568] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7889.447310] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7897.837052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7898.182619] Lustre: lustre-OST0000: new disk, initializing [ 7898.187942] Lustre: Skipped 1 previous similar message [ 7898.192281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7898.200302] Lustre: Skipped 2 previous similar messages [ 7898.208566] Lustre: 172610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7904.859127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7904.880457] Lustre: Skipped 1 previous similar message [ 7904.885419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7905.187138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7906.809723] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7918.142949] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7924.279731] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7924.505453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7930.397813] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7930.745842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7935.974811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7935.979356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.999817] LustreError: Skipped 1 previous similar message [ 7936.006525] Lustre: Skipped 1 previous similar message [ 7936.014337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7936.039549] Lustre: Skipped 1 previous similar message [ 7939.781587] Lustre: server umount lustre-OST0000 complete [ 7951.335956] LustreError: 170649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7951.355332] Lustre: server umount lustre-MDT0000 complete [ 7951.363288] LustreError: 170649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 32 previous similar messages [ 7956.620523] LustreError: 170634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786109365 with bad export cookie 7211968194491857891 [ 7956.636926] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7956.976262] Lustre: server umount lustre-MDT0001 complete [ 7967.941933] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 09:29:34 (1786109374) [ 7976.584700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.847482] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7992.361266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.204279] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8007.249291] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8013.556875] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8021.013266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8029.484695] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8039.211813] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8054.331867] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 09:31:00 (1786109460) [ 8063.469559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8063.470528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8063.486140] Lustre: Skipped 4 previous similar messages [ 8063.487240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8063.492576] LustreError: Skipped 1 previous similar message [ 8063.533144] Lustre: Skipped 3 previous similar messages [ 8064.955772] Lustre: server umount lustre-OST0000 complete [ 8075.413495] Lustre: server umount lustre-MDT0000 complete [ 8080.008311] LustreError: 175006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786109488 with bad export cookie 7211968194491858773 [ 8080.018965] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8080.020973] LustreError: 175006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8080.539350] Lustre: server umount lustre-MDT0001 complete [ 8094.135090] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8097.385975] Key type lgssc unregistered [ 8097.816329] LNet: 178756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8097.830558] LNetError: 178756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8098.869488] LNet: Removed LNI 192.168.204.142@tcp [ 8099.802431] Key type .llcrypt unregistered [ 8099.805504] Key type ._llcrypt unregistered [ 8121.484517] Key type ._llcrypt registered [ 8121.487403] Key type .llcrypt registered [ 8121.609711] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8123.684397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8123.827247] alg: No test for adler32 (adler32-zlib) [ 8125.246650] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 8125.580196] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 8127.383398] Key type lgssc registered [ 8129.163323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8138.132428] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8138.177299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8153.823397] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8158.943501] LustreError: 180001:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 8164.064081] LustreError: 180001:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8164.080622] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8164.088762] LustreError: MGC192.168.204.142@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8164.103594] LustreError: 180001:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8164.120463] LustreError: 180001:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8164.130715] LustreError: 180001:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8164.194395] Lustre: server umount lustre-OST0000 complete [ 8164.197655] LustreError: 180001:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8173.906206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8175.254274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.615680] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8184.802397] LustreError: 180500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8184.823833] LustreError: 180500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8189.934954] LustreError: 180501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8191.459224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8191.825971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8198.030642] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8207.062670] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8213.778719] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8220.911761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.264766] LustreError: 182316:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8221.271966] Lustre: 182316:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.142@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8221.532041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.510339] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8238.941934] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8243.173148] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8243.182553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8243.184733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8243.196860] Lustre: Skipped 2 previous similar messages [ 8243.226804] Lustre: Skipped 2 previous similar messages [ 8247.975297] Lustre: server umount lustre-MDT0000 complete [ 8248.304698] LustreError: 181435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8248.351954] LustreError: 181435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8253.306444] LustreError: 180481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786109662 with bad export cookie 12139581316857421753 [ 8253.312882] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8253.336240] LustreError: 180481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8253.412938] LustreError: 180497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.423761] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.458977] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8253.755048] Lustre: server umount lustre-MDT0001 complete [ 8260.405542] Lustre: server umount lustre-OST0000 complete [ 8271.976808] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8283.634827] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8325.946945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8326.207236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8326.253766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8326.371359] Lustre: lustre-MDT0000: new disk, initializing [ 8326.438277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8326.458669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8332.689698] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8344.846721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.980621] Lustre: 185842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8345.037162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8345.040746] Lustre: Skipped 1 previous similar message [ 8345.102671] Lustre: lustre-MDT0001: new disk, initializing [ 8345.249347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8345.308223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8345.317900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8351.153467] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8362.518355] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8370.216928] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8379.795446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8380.444987] Lustre: lustre-OST0000: new disk, initializing [ 8380.455869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8380.468690] Lustre: 187808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8380.645274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8386.164483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8386.185244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8386.399607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8388.964372] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8400.892783] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8408.293775] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8408.668193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8414.242226] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8414.526541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8417.266743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8417.280237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8417.300049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8417.309988] Lustre: Skipped 1 previous similar message [ 8422.387819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8422.396392] Lustre: Skipped 1 previous similar message [ 8423.217773] Lustre: server umount lustre-OST0000 complete [ 8427.499581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8427.505237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8427.509394] Lustre: Skipped 1 previous similar message [ 8427.511521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8427.522551] LustreError: Skipped 1 previous similar message [ 8433.980494] Lustre: server umount lustre-MDT0000 complete [ 8437.731758] LustreError: 185847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8437.779313] LustreError: 185847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8438.843619] LustreError: 185832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786109847 with bad export cookie 12139581316857422467 [ 8438.852716] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8438.866641] LustreError: 185832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8439.198535] Lustre: server umount lustre-MDT0001 complete [ 8450.269644] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 09:37:36 (1786109856) [ 8473.517250] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8476.999662] Key type lgssc unregistered [ 8477.523679] LNet: 190765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8477.533536] LNetError: 190765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8478.567294] LNet: Removed LNI 192.168.204.142@tcp [ 8479.694623] Key type .llcrypt unregistered [ 8479.702291] Key type ._llcrypt unregistered [ 8505.929249] Key type ._llcrypt registered [ 8505.932185] Key type .llcrypt registered [ 8506.065531] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8508.117285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8508.321166] alg: No test for adler32 (adler32-zlib) [ 8510.320752] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 8510.937173] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 8512.799148] Key type lgssc registered [ 8514.626481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8525.600688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8525.649251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.081141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8531.742776] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8536.555035] LustreError: 191989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8536.613649] LustreError: 191989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8541.684312] LustreError: 191988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8542.205520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.574549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8547.511579] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8556.150522] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8563.763481] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8575.114991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8575.252506] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8575.254983] LustreError: 193853:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8575.261183] Lustre: 193853:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.142@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8575.355802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8583.070572] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8591.989045] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8597.989632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8597.997519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8598.008252] Lustre: Skipped 2 previous similar messages [ 8598.034076] Lustre: Skipped 2 previous similar messages [ 8600.578149] Lustre: server umount lustre-MDT0000 complete [ 8603.108758] LustreError: 191988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8603.139085] LustreError: 191988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8605.061664] LustreError: 191969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110013 with bad export cookie 12135545636577571629 [ 8605.089716] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8605.094156] LustreError: 191969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8605.561889] Lustre: server umount lustre-MDT0001 complete [ 8620.952631] Lustre: server umount lustre-OST0000 complete [ 8632.782192] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8645.142938] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8686.875030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.134151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8687.194704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8687.346713] Lustre: lustre-MDT0000: new disk, initializing [ 8687.478266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8687.516277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8693.470763] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8705.898478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8706.172602] Lustre: 197379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8706.285101] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8706.288803] Lustre: Skipped 1 previous similar message [ 8706.292705] LustreError: 198161:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8706.307465] LustreError: 198161:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8706.315686] Lustre: 198161:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8706.351520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8706.359077] Lustre: Skipped 1 previous similar message [ 8706.500491] Lustre: lustre-MDT0001: new disk, initializing [ 8706.687372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8706.820429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8706.842348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8712.439405] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8723.826523] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8730.618889] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8738.569767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8738.980200] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8738.986608] Lustre: Skipped 1 previous similar message [ 8738.988875] LustreError: 199321:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8739.004511] LustreError: 199321:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8739.012837] Lustre: 199321:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8739.028079] Lustre: 199321:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8739.062794] Lustre: lustre-OST0000: new disk, initializing [ 8739.071390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8739.082376] Lustre: 199344:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8739.243040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8744.626321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8744.636212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8744.779121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8747.723590] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8757.559233] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8764.502359] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8764.853364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8771.900257] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8772.236625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8775.649561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8775.665870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8775.698816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8775.711772] Lustre: Skipped 1 previous similar message [ 8780.770088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8780.786224] Lustre: Skipped 1 previous similar message [ 8781.188990] Lustre: server umount lustre-OST0000 complete [ 8785.890876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8785.908344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8785.929977] Lustre: Skipped 2 previous similar messages [ 8791.014170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8791.031387] Lustre: Skipped 2 previous similar messages [ 8791.994289] Lustre: server umount lustre-MDT0000 complete [ 8796.133693] LustreError: 197390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8796.147419] LustreError: 197390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8797.017149] LustreError: 200888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110205 with bad export cookie 12135545636577572336 [ 8797.027333] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8797.539669] Lustre: server umount lustre-MDT0001 complete [ 8808.316621] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 09:43:34 (1786110214) [ 8828.179137] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8841.386387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8841.697651] LustreError: 202481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8841.753426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8847.069962] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8851.940848] LustreError: 202482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8851.980343] LustreError: 202482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8855.410510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8860.452329] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8863.572809] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8869.922130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8878.837666] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8890.608686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8890.913274] Lustre: lustre-OST0001: new disk, initializing [ 8890.921794] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8890.927633] Lustre: 204788:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8891.048258] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8891.053854] Lustre: Skipped 2 previous similar messages [ 8898.006742] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8899.310198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8899.336568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8899.467996] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8910.251284] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8919.303352] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8921.976981] Lustre: DEBUG MARKER: ADD OST3 [ 8927.077784] loop: module loaded [ 8927.175904] loop0: detected capacity change from 0 to 204800000 [ 8927.468474] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8927.480912] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8927.517242] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8934.320624] loop0: detected capacity change from 0 to 204800000 [ 8935.031831] Lustre: DEBUG MARKER: STOP OST3 [ 8941.127289] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8941.493132] Lustre: lustre-OST0003: new disk, initializing [ 8941.502574] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8941.509739] Lustre: 206829:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8942.964293] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8943.169881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8943.178134] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8943.237732] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8944.993804] LustreError: 206946:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8945.027542] Lustre: Failing over lustre-OST0003 [ 8945.157645] Lustre: server umount lustre-OST0003 complete [ 8948.216444] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8948.217470] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8948.231815] LustreError: 203965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.244817] Lustre: Skipped 1 previous similar message [ 8948.290648] LustreError: 203965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8949.260307] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8961.284350] Lustre: DEBUG MARKER: START OST3 again [ 8967.847667] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8968.241972] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8968.258432] Lustre: Skipped 1 previous similar message [ 8968.290763] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8969.004964] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8969.956372] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8969.957201] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8975.297503] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8996.647256] Lustre: Failing over lustre-OST0003 [ 8996.759990] Lustre: server umount lustre-OST0003 complete [ 8998.899760] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8998.907286] LustreError: 203965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8998.915831] Lustre: Skipped 1 previous similar message [ 8998.944442] LustreError: 203965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 9014.758486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9014.769237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9014.789099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9020.424041] Lustre: server umount lustre-MDT0000 complete [ 9023.412119] LustreError: 202463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110432 with bad export cookie 12135545636577573225 [ 9023.415770] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9023.428800] LustreError: 202463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9029.744273] Lustre: server umount lustre-MDT0001 complete [ 9033.431963] Lustre: server umount lustre-OST0000 complete [ 9037.101627] Lustre: server umount lustre-OST0001 complete [ 9043.511663] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9053.083436] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9088.822403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9089.017248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9089.066753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9089.132393] Lustre: lustre-MDT0000: new disk, initializing [ 9089.245530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9094.300739] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9105.307294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9105.448261] Lustre: 212400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9105.541443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9105.545567] Lustre: Skipped 1 previous similar message [ 9105.661593] Lustre: lustre-MDT0001: new disk, initializing [ 9105.759703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9105.776323] Lustre: Skipped 1 previous similar message [ 9105.838776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9105.853304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9110.258468] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9119.446228] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9125.407910] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9132.786824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9133.222272] Lustre: 214370:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9135.394809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9135.665204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9140.601895] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9152.047594] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9156.342360] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9156.555501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9161.166572] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9161.492711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9166.305415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9166.307819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9166.316627] LustreError: Skipped 1 previous similar message [ 9166.325193] Lustre: Skipped 5 previous similar messages [ 9166.326953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9166.346866] Lustre: Skipped 9 previous similar messages [ 9169.504974] Lustre: server umount lustre-OST0000 complete [ 9179.464894] Lustre: server umount lustre-MDT0000 complete [ 9181.672266] LustreError: 212405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9181.689735] LustreError: 212405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 9182.820898] LustreError: 212390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110591 with bad export cookie 12135545636577575521 [ 9182.823602] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9182.834557] LustreError: 212390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9183.153856] Lustre: server umount lustre-MDT0001 complete [ 9186.432220] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 09:49:53 (1786110593) [ 9201.552155] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9211.841559] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9244.612832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9244.751562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9244.777215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9244.780164] Lustre: Skipped 1 previous similar message [ 9244.882975] Lustre: lustre-MDT0000: new disk, initializing [ 9244.886128] Lustre: Skipped 1 previous similar message [ 9244.971928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9244.978487] Lustre: Skipped 1 previous similar message [ 9248.990600] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9258.410284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9258.532875] Lustre: 218982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9258.780060] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9262.563862] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9271.747198] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9276.815855] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9282.499842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9282.759069] Lustre: 220950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9284.530536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9287.289474] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9295.324806] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9299.179702] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9299.379548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9303.241114] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9303.488619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9310.179570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9310.180669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9310.191677] Lustre: Skipped 3 previous similar messages [ 9310.197958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9310.209704] Lustre: Skipped 3 previous similar messages [ 9319.391546] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9319.527233] Lustre: server umount lustre-OST0000 complete [ 9325.539437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9325.548383] LustreError: Skipped 1 previous similar message [ 9328.880078] Lustre: server umount lustre-MDT0000 complete [ 9330.663961] LustreError: 218993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9330.694373] LustreError: 218993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9331.834464] LustreError: 218972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110740 with bad export cookie 12135545636577576403 [ 9331.842319] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9331.875858] LustreError: 218972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9332.225176] Lustre: server umount lustre-MDT0001 complete [ 9338.535600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9342.243275] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9349.751453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9353.799809] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9359.595154] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9363.960404] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9369.033992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9369.273866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9369.283565] Lustre: Skipped 6 previous similar messages [ 9374.552433] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9380.416511] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9395.168147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9395.172885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9395.182066] Lustre: Skipped 2 previous similar messages [ 9395.189472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9395.195834] Lustre: Skipped 4 previous similar messages [ 9400.571409] Lustre: server umount lustre-OST0000 complete [ 9409.470838] Lustre: server umount lustre-MDT0000 complete [ 9412.238854] LustreError: 225866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110820 with bad export cookie 12135545636577577285 [ 9412.242957] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9412.248187] LustreError: 225866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9412.465583] Lustre: server umount lustre-MDT0001 complete [ 9516.426582] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 9518.658444] Key type lgssc unregistered [ 9518.907433] LNet: 226856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9518.915532] LNetError: 226856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9518.926794] LNet: Removed LNI 192.168.204.142@tcp [ 9519.613186] Key type .llcrypt unregistered [ 9519.615572] Key type ._llcrypt unregistered [ 9538.345738] Key type ._llcrypt registered [ 9538.350139] Key type .llcrypt registered [ 9538.449373] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9550.266982] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9551.020225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9551.105276] alg: No test for adler32 (adler32-zlib) [ 9552.255979] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 9552.437791] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 9554.095240] Key type lgssc registered [ 9555.214862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9581.706812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9581.758442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9582.913830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9582.950729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9583.014619] Lustre: lustre-MDT0000: new disk, initializing [ 9583.087927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9583.138681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9586.521747] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9594.731164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9594.849941] Lustre: 230185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9594.909970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9594.913536] Lustre: Skipped 1 previous similar message [ 9594.980759] Lustre: lustre-MDT0001: new disk, initializing [ 9595.057470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9595.104566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9595.116029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9598.659727] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9605.589758] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9609.599428] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9614.979968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9615.189341] Lustre: lustre-OST0000: new disk, initializing [ 9615.192727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9615.202309] Lustre: 232153:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9615.302257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9619.905369] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9621.552842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9621.561935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9621.642866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9626.384236] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9629.663869] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9629.789330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9633.006626] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9633.177617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9636.832608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9636.836785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9636.844484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9637.348568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9637.351864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9637.369015] Lustre: Skipped 1 previous similar message [ 9640.818619] Lustre: server umount lustre-OST0000 complete [ 9646.560628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9646.567220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9646.576154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9652.703995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9652.710309] Lustre: Skipped 3 previous similar messages [ 9657.824525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9657.829160] Lustre: Skipped 1 previous similar message [ 9658.335517] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9658.461385] Lustre: server umount lustre-MDT0000 complete [ 9661.315930] LustreError: 232163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111070 with bad export cookie 15654212209785170702 [ 9661.321533] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9661.324351] LustreError: 232163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9661.542388] Lustre: server umount lustre-MDT0001 complete [ 9675.451513] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9682.889067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9683.185408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9686.504370] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9688.546530] LustreError: 235085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9688.569897] LustreError: 235085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9692.425409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9692.653466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9695.721308] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9698.021775] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9702.847164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9703.163665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9708.310069] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9714.573885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9714.698911] Lustre: lustre-OST0001: new disk, initializing [ 9714.707123] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9714.711371] Lustre: 237394:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9717.853700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9717.867298] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9717.967494] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9719.412758] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9728.670787] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9736.417591] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9745.397799] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 09:59:12 (1786111152) [ 9748.963074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9748.968462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.978111] Lustre: Skipped 1 previous similar message [ 9748.985791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9752.914714] Lustre: server umount lustre-MDT0000 complete [ 9755.107980] LustreError: 235811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9755.125674] LustreError: 235811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9755.473588] LustreError: 237392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111164 with bad export cookie 15654212209785171584 [ 9755.474833] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9755.481276] LustreError: 237392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9755.723065] Lustre: server umount lustre-MDT0001 complete [ 9768.492737] Lustre: server umount lustre-OST0000 complete [ 9781.025574] Lustre: server umount lustre-OST0001 complete [ 9785.749307] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9792.587025] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9817.993321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9818.137215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9818.165321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9818.231274] Lustre: lustre-MDT0000: new disk, initializing [ 9818.315432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9818.323284] Lustre: Skipped 1 previous similar message [ 9818.346323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9821.363301] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9829.347919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9829.460108] Lustre: 241333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9829.509284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9829.512223] Lustre: Skipped 1 previous similar message [ 9829.570269] Lustre: lustre-MDT0001: new disk, initializing [ 9829.691246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9829.702945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9832.842656] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9839.583188] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9843.255728] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9847.515332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9847.698952] Lustre: lustre-OST0000: new disk, initializing [ 9847.702437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9847.707581] Lustre: 243302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9849.686490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9849.695508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9849.770598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9851.575456] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9858.580374] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9861.942518] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9862.207512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9865.416456] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9865.589901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9870.306561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9870.311092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9870.314479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9870.314487] Lustre: Skipped 4 previous similar messages [ 9870.314488] Lustre: Skipped 4 previous similar messages [ 9881.055090] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9881.153892] Lustre: server umount lustre-OST0000 complete [ 9885.666385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9885.676199] Lustre: Skipped 1 previous similar message [ 9889.979423] Lustre: server umount lustre-MDT0000 complete [ 9890.789350] LustreError: 242218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9890.812193] LustreError: 242218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9892.625818] LustreError: 243301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111301 with bad export cookie 15654212209785173719 [ 9892.629051] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9892.635742] LustreError: 243301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9892.848812] Lustre: server umount lustre-MDT0001 complete [ 9898.853573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9899.134273] LustreError: 245527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9899.226806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9899.232447] Lustre: Skipped 2 previous similar messages [ 9902.263930] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9904.609575] LustreError: 245528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9907.841622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9911.075931] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9915.887923] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9919.454734] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9923.632293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9927.944083] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9932.379830] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10081.760611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10081.770162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10081.773115] LustreError: Skipped 2 previous similar messages [10081.779379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10081.783927] Lustre: Skipped 8 previous similar messages [10084.576928] Lustre: server umount lustre-OST0000 complete [10094.996992] Lustre: server umount lustre-MDT0000 complete [10097.129698] LustreError: 245528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10097.222202] LustreError: 245528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10099.850321] LustreError: 245507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111508 with bad export cookie 15654212209785174601 [10099.862035] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10099.878747] LustreError: 245507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10100.390573] Lustre: server umount lustre-MDT0001 complete [10205.422233] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [10208.975262] Key type lgssc unregistered [10209.277821] LNet: 249137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10209.287898] LNetError: 249137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10209.299725] LNet: Removed LNI 192.168.204.142@tcp [10210.629566] Key type .llcrypt unregistered [10210.633811] Key type ._llcrypt unregistered [10244.661850] Key type ._llcrypt registered [10244.669677] Key type .llcrypt registered [10244.900528] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [10264.759227] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10265.492211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10265.525031] alg: No test for adler32 (adler32-zlib) [10268.008537] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [10268.274485] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [10270.055159] Key type lgssc registered [10271.963835] Lustre: Echo OBD driver; http://www.lustre.org/ [10324.493565] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10335.674041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10335.725880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10336.940236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10336.971232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10337.054072] Lustre: lustre-MDT0000: new disk, initializing [10337.137649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10337.159610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10342.137241] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10354.439639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10354.578619] Lustre: 253210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10354.626788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10354.630323] Lustre: Skipped 1 previous similar message [10354.682598] Lustre: lustre-MDT0001: new disk, initializing [10354.809499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10354.878979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10354.888507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10359.639672] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10364.979432] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10371.863654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10372.455070] Lustre: lustre-OST0000: new disk, initializing [10372.464662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10372.483853] Lustre: 254840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10372.625579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10379.336468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10379.364684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10379.647530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10380.450557] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10392.767731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10392.969270] Lustre: lustre-OST0001: new disk, initializing [10392.973347] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10392.980182] Lustre: 255713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10393.118785] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10401.012408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10401.042349] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10401.335226] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10402.819745] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10416.973470] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10425.095713] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10437.095064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10437.110251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10437.132299] Lustre: Skipped 1 previous similar message [10442.214370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10442.222014] Lustre: Skipped 6 previous similar messages [10443.508339] Lustre: server umount lustre-MDT0000 complete [10448.396808] LustreError: 254838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111857 with bad export cookie 17395447882879735122 [10448.406220] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10448.419199] LustreError: 254838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10448.779588] Lustre: server umount lustre-MDT0001 complete [10463.706194] Lustre: server umount lustre-OST0000 complete [10477.536100] Lustre: server umount lustre-OST0001 complete [10494.345915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10506.234023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10515.790965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10527.156993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10551.346342] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10562.485863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10562.510523] Lustre: lustre-MDT0000: reset Object Index mappings [10562.896758] LustreError: 260726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10562.913051] LustreError: 260726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10562.996887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10567.836758] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10568.170563] LustreError: 260727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10573.288891] LustreError: 260726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10577.327220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10577.342785] Lustre: lustre-MDT0001: reset Object Index mappings [10577.739835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10583.757539] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10587.470903] Lustre: 261869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10594.757747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10595.276403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10602.489899] LustreError: 262223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10604.907872] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10607.592232] LustreError: 262524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10608.668374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10616.608847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10622.470696] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10624.234820] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10633.021770] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10637.083205] Lustre: 263740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10642.927861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10642.931143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10642.949201] Lustre: Skipped 4 previous similar messages [10642.967094] Lustre: Skipped 1 previous similar message [10643.615041] Lustre: server umount lustre-MDT0000 complete [10647.354210] LustreError: 260707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112056 with bad export cookie 17395447882879741912 [10647.358129] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10647.387166] LustreError: 260707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10647.812526] Lustre: server umount lustre-MDT0001 complete [10661.734948] Lustre: server umount lustre-OST0000 complete [10663.651204] Lustre: 250717:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786112056/real 1786112056] req@ffff9706cbf01500 x1872873836907904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786112072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10663.717802] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10663.734372] Lustre: Skipped 1 previous similar message [10665.647657] Lustre: server umount lustre-OST0001 complete [10688.069564] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:14:54 (1786112094) [10706.870280] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [10719.823726] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10768.668261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10768.921839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10768.953200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10769.028700] Lustre: lustre-MDT0000: new disk, initializing [10769.124996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10769.131386] Lustre: Skipped 1 previous similar message [10769.155718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10773.828885] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10785.959570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10786.147439] Lustre: 267998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10786.209391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10786.211825] Lustre: Skipped 1 previous similar message [10786.281609] Lustre: lustre-MDT0001: new disk, initializing [10786.453646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10786.474823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10792.413717] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10806.368889] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10815.716718] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10823.916822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10824.178728] Lustre: lustre-OST0000: new disk, initializing [10824.181981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10824.189405] Lustre: 269968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10825.441257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10825.467910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10825.673710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10833.077934] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10846.873191] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10853.106235] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10853.344722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10859.330730] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10859.868040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10866.662895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10866.665431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10866.688650] LustreError: Skipped 1 previous similar message [10866.706083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10866.710413] Lustre: Skipped 1 previous similar message [10866.739746] Lustre: Skipped 1 previous similar message [10868.657949] Lustre: server umount lustre-OST0000 complete [10873.314788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10873.327301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10873.344674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10879.309832] Lustre: server umount lustre-MDT0000 complete [10882.025156] LustreError: 268847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10882.063241] LustreError: 268847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10883.446816] LustreError: 267989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112292 with bad export cookie 17395447882879744005 [10883.458753] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10883.470042] LustreError: 267989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10883.799885] Lustre: server umount lustre-MDT0001 complete [10906.898978] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10917.796074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10918.066035] LustreError: 272904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.110881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10918.115140] Lustre: Skipped 2 previous similar messages [10922.235787] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10930.443302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10935.568553] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10938.409188] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10944.404439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10950.970247] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10959.286884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10959.549915] Lustre: lustre-OST0001: new disk, initializing [10959.553743] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10959.569033] Lustre: 275211:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10961.418956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10961.429879] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10961.595745] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10965.841214] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10976.323838] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10980.364951] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10985.462626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10985.557459] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10992.104087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10992.117597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.136737] Lustre: Skipped 1 previous similar message [10992.150581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10992.159656] Lustre: Skipped 2 previous similar messages [10997.988068] Lustre: server umount lustre-MDT0000 complete [10998.758333] LustreError: 272899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10998.769136] LustreError: 272899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11001.121683] LustreError: 272884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112409 with bad export cookie 17395447882879744887 [11001.128688] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11001.134545] LustreError: 272884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11001.388842] Lustre: server umount lustre-MDT0001 complete [11015.391313] Lustre: server umount lustre-OST0000 complete [11029.434560] Lustre: server umount lustre-OST0001 complete [11046.574482] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11057.415264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11057.714147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11057.724286] Lustre: Skipped 3 previous similar messages [11062.418623] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11067.877493] LustreError: 278270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11067.899586] LustreError: 278270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11072.380382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11077.728501] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11081.297758] Lustre: 279409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11088.268634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11095.441178] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11103.005359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11104.695823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11110.424463] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11111.254820] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11120.253496] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11124.418563] Lustre: 281287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11131.298167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11131.480195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11143.778984] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:22:29 (1786112549) [11161.581302] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11161.590049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11161.605378] Lustre: Skipped 6 previous similar messages [11161.622443] Lustre: Skipped 7 previous similar messages [11175.391244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11175.689299] Lustre: server umount lustre-MDT0000 complete [11179.822951] LustreError: 278250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112588 with bad export cookie 17395447882879751418 [11179.840771] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11179.843234] LustreError: 278250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11194.335331] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11194.641765] Lustre: server umount lustre-MDT0001 complete [11213.279163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11213.374441] Lustre: server umount lustre-OST0000 complete [11217.679926] Lustre: server umount lustre-OST0001 complete [11225.092467] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [11236.727106] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11286.200373] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11297.688295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11297.842576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11297.871195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11297.926859] Lustre: lustre-MDT0000: new disk, initializing [11298.001455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11302.539733] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11313.521496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11313.693667] Lustre: 285532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11314.019737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11314.035995] Lustre: Skipped 4 previous similar messages [11314.118761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11318.886866] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11329.993702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11330.282986] Lustre: 287166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11331.969672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11336.846562] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11346.418502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11346.558059] Lustre: 288038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11348.036784] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11348.048945] Lustre: Skipped 1 previous similar message [11348.168324] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11351.976116] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11361.787400] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11365.481301] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11365.484338] Lustre: Skipped 1 previous similar message [11379.807483] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:26:26 (1786112786) [11384.173211] Lustre: server umount lustre-MDT0000 complete [11384.296067] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11384.300985] LustreError: 285544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11384.323325] Lustre: Skipped 3 previous similar messages [11384.357205] LustreError: 285544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11387.363647] LustreError: 285524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112796 with bad export cookie 17395447882879759104 [11387.367124] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11387.369679] LustreError: 285524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11387.676362] Lustre: server umount lustre-MDT0001 complete [11401.280432] Lustre: server umount lustre-OST0000 complete [11415.033216] Lustre: server umount lustre-OST0001 complete [11424.388515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11429.717365] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11438.346946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11444.118761] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11451.081434] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11456.748796] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11463.598901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11470.124399] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11477.997788] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11497.230331] loop1: detected capacity change from 0 to 204800000 [11497.368209] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11497.380934] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11497.407082] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11502.402961] loop1: detected capacity change from 0 to 204800000 [11507.226873] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11507.278517] LustreError: 290271:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs29749: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11507.286148] LustreError: 290271:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs29749 [11507.290350] LustreError: 290271:0:(mgs_handler.c:517:mgs_target_reg()) fs29749-MDT0001: Error checking target index: rc = -13 [11507.296665] LustreError: fs29749-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11507.306879] LustreError: 294003:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11507.312537] LustreError: 294003:0:(tgt_mount.c:2129:server_put_super()) no obd fs29749-MDT0001 [11507.316592] LustreError: 294003:0:(tgt_mount.c:116:server_deregister_mount()) fs29749-MDT0001 not registered [11507.339270] Lustre: server umount fs29749-MDT0001 complete [11507.342972] LustreError: 294003:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11513.580433] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11513.645675] Lustre: Setting parameter fs29749-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs29749-MDT0001 [11513.688164] Lustre: srv-fs29749-MDT0001: No data found on store. Initialize space. [11513.691168] Lustre: Skipped 4 previous similar messages [11513.771692] Lustre: fs29749-MDT0001: new disk, initializing [11513.775430] Lustre: Skipped 3 previous similar messages [11513.872832] Lustre: srv-fs29749-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11513.882682] Lustre: 294469:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29749-MDT0001-mdtlov: can't create new llog: rc = -115 [11513.890919] LustreError: 294469:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29749-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11517.050833] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11522.501630] Lustre: server umount fs29749-MDT0001 complete [11533.699858] loop1: detected capacity change from 0 to 204800000 [11538.139506] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11538.410408] Lustre: 296036:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29749-MDT0001-mdtlov: can't create new llog: rc = -115 [11538.417911] LustreError: 296036:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29749-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11541.841509] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11545.814336] Lustre: server umount fs29749-MDT0001 complete [11550.193502] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11555.296840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11555.303750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11555.308175] Lustre: Skipped 4 previous similar messages [11555.324269] Lustre: Skipped 18 previous similar messages [11559.090857] Lustre: server umount lustre-MDT0000 complete [11565.551063] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11565.896288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11565.990228] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11569.578931] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11571.691155] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11575.495095] loop1: detected capacity change from 0 to 204800000 [11579.639327] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11579.936829] Lustre: 298481:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29749-MDT0001-mdtlov: can't create new llog: rc = -115 [11579.963189] LustreError: 298481:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29749-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11583.521626] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11586.855393] Lustre: server umount fs29749-MDT0001 complete [11608.379782] Lustre: server umount lustre-OST0000 complete [11612.648696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11616.964641] Lustre: server umount lustre-MDT0000 complete [11619.344672] LustreError: 290263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786113028 with bad export cookie 17395447882879761785 [11619.350355] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11619.356762] LustreError: 290263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11619.629626] Lustre: server umount lustre-MDT0001 complete [11627.138197] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11628.965809] Key type lgssc unregistered [11629.183101] LNet: 300828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11629.191968] LNetError: 300828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11630.245820] LNet: Removed LNI 192.168.204.142@tcp [11630.762350] Key type .llcrypt unregistered [11630.765313] Key type ._llcrypt unregistered [11643.590900] Key type ._llcrypt registered [11643.593248] Key type .llcrypt registered [11643.660040] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11644.212541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11644.224899] alg: No test for adler32 (adler32-zlib) [11645.176861] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [11645.330283] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [11646.967196] Key type lgssc registered [11647.679643] Lustre: Echo OBD driver; http://www.lustre.org/ [11652.607901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11652.618613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11653.896368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11656.607867] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11662.217990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11662.506170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11665.385482] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11670.112673] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11673.898817] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11678.600781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11678.820498] LustreError: 303908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.831208] LustreError: 303908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11678.881748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11682.915624] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11684.325383] LustreError: 303905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11687.282865] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11687.399865] LustreError: 303907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11687.411291] LustreError: 303907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11689.710434] LustreError: 303905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.884406] LustreError: 304577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.895986] LustreError: 304577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11700.457313] loop1: detected capacity change from 0 to 204800000 [11700.594057] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11700.607286] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11700.614491] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11704.583715] loop1: detected capacity change from 0 to 204800000 [11705.125588] LustreError: 303908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11705.144094] LustreError: 303908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11708.530104] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11708.640795] Lustre: fs23616-OST0000: new disk, initializing [11708.646057] Lustre: srv-fs23616-OST0000: No data found on store. Initialize space. [11708.651629] Lustre: 305811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11708.735476] Lustre: fs23616-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11713.119825] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11719.224957] Lustre: server umount fs23616-OST0000 complete [11723.237384] LustreError: 304577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11723.250899] LustreError: 304577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11726.779613] loop1: detected capacity change from 0 to 204800000 [11730.115413] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11730.240294] Lustre: fs23616-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11734.276500] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11739.829201] loop2: detected capacity change from 0 to 204800000 [11739.940830] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11739.950128] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11739.963886] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11743.154750] loop2: detected capacity change from 0 to 204800000 [11746.255830] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11746.304147] LustreError: fs23616-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11746.309687] LustreError: fs23616-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11746.315595] LustreError: 308850:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11746.320334] LustreError: 308850:0:(tgt_mount.c:2129:server_put_super()) no obd fs23616-OST0001 [11746.324273] LustreError: 308850:0:(tgt_mount.c:116:server_deregister_mount()) fs23616-OST0001 not registered [11746.344871] Lustre: server umount fs23616-OST0001 complete [11746.347433] LustreError: 308850:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11751.797650] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11751.889760] Lustre: fs23616-OST0001: new disk, initializing [11751.892955] Lustre: srv-fs23616-OST0001: No data found on store. Initialize space. [11751.898223] Lustre: 309401:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11751.949113] Lustre: fs23616-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11755.191948] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11756.323880] LustreError: 303905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11756.334842] LustreError: 303905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11760.619878] Lustre: server umount fs23616-OST0000 complete [11764.202118] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11765.650844] Lustre: server umount fs23616-OST0001 complete [11772.747418] loop1: detected capacity change from 0 to 204800000 [11774.442866] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11775.861707] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11775.980595] Lustre: fs23616-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11779.444770] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11783.585959] loop2: detected capacity change from 0 to 204800000 [11786.753443] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11790.389599] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11796.366147] Lustre: server umount fs23616-OST0000 complete [11802.019653] Lustre: server umount fs23616-OST0001 complete [11810.277626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11810.278326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11810.283291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11810.291961] Lustre: Skipped 1 previous similar message [11810.302134] Lustre: Skipped 1 previous similar message [11815.392687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11815.399634] Lustre: Skipped 1 previous similar message [11815.533803] Lustre: server umount lustre-OST0000 complete [11820.514167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11820.516815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11820.522426] Lustre: Skipped 1 previous similar message [11820.530910] Lustre: Skipped 1 previous similar message [11825.632577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11825.636965] Lustre: Skipped 1 previous similar message [11830.752474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11830.754650] Lustre: Skipped 1 previous similar message [11832.287254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11832.402536] Lustre: server umount lustre-MDT0000 complete [11834.463814] LustreError: 302060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786113243 with bad export cookie 3866170690801918211 [11834.469265] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11834.469830] LustreError: 302060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11834.687902] Lustre: server umount lustre-MDT0001 complete [11840.104198] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11841.611591] Key type lgssc unregistered [11841.777553] LNet: 315321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11841.782792] LNetError: 315321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11841.798445] LNet: Removed LNI 192.168.204.142@tcp [11842.294179] Key type .llcrypt unregistered [11842.296769] Key type ._llcrypt unregistered [11860.392486] Key type ._llcrypt registered [11860.394789] Key type .llcrypt registered [11860.475504] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [11870.336246] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11870.810843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11870.873572] alg: No test for adler32 (adler32-zlib) [11871.822872] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [11871.954896] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [11873.567392] Key type lgssc registered [11874.211709] Lustre: Echo OBD driver; http://www.lustre.org/ [11894.712244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11894.723767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11895.848119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11895.870623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11895.923759] Lustre: lustre-MDT0000: new disk, initializing [11895.959152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11895.971246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11898.202417] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11904.542664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11904.605107] Lustre: 319153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11904.638996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11904.641651] Lustre: Skipped 1 previous similar message [11904.685559] Lustre: lustre-MDT0001: new disk, initializing [11904.742695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11904.766137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11904.772986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11907.199520] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11912.768454] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11916.022336] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11919.528720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11919.668669] Lustre: lustre-OST0000: new disk, initializing [11919.671664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11919.676347] Lustre: 321120:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11919.720798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11923.098521] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11928.909862] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11929.105747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11929.113541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11929.181711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11930.024661] Lustre: Mounted lustre-client - version 2.17.56_50_gf5ff674 [11930.600528] Lustre: Unmounted lustre-client [11939.297614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11939.299737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11939.305399] LustreError: Skipped 1 previous similar message [11939.315738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11939.324465] Lustre: Skipped 1 previous similar message [11940.324579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11940.329542] Lustre: Skipped 1 previous similar message [11942.088266] Lustre: server umount lustre-OST0000 complete [11945.443260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11945.445186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11945.456601] Lustre: Skipped 2 previous similar messages [11945.466446] Lustre: Skipped 1 previous similar message [11950.566079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11950.569551] Lustre: Skipped 1 previous similar message [11950.630817] Lustre: server umount lustre-MDT0000 complete [11953.108544] LustreError: 319947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786113361 with bad export cookie 622947012857017374 [11953.113184] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11953.122353] LustreError: 319947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11953.295653] Lustre: server umount lustre-MDT0001 complete [11960.091789] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11961.714190] Key type lgssc unregistered [11990.560241] LNet: 323285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11990.566562] LNetError: 323285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12060.197421] LNet: Removed LNI 192.168.204.142@tcp [12060.756418] Key type .llcrypt unregistered [12060.758656] Key type ._llcrypt unregistered [12070.643374] Key type ._llcrypt registered [12070.645140] Key type .llcrypt registered [12070.721334] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12079.867432] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12080.449668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12080.493502] alg: No test for adler32 (adler32-zlib) [12081.391660] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [12081.532372] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12083.135179] Key type lgssc registered [12083.814682] Lustre: Echo OBD driver; http://www.lustre.org/ [12099.804926] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 10:38:27 (1786113507) [12103.305027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12103.336880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12104.490703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12104.514184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12104.562788] Lustre: lustre-MDT0000: new disk, initializing [12104.609404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12104.619485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12106.786212] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12112.819129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12112.879408] Lustre: 326485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12112.929308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12112.931822] Lustre: Skipped 1 previous similar message [12112.980420] Lustre: lustre-MDT0001: new disk, initializing [12113.038982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12113.073542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12113.095482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12115.171193] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12120.119973] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12123.223266] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12126.939981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12127.081475] Lustre: lustre-OST0000: new disk, initializing [12127.084241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12127.088935] Lustre: 328452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12127.140520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12129.975950] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12135.416837] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12136.979185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12136.990705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12137.045145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12150.433718] Lustre: server umount lustre-OST0000 complete [12152.289082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12152.290527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12152.295182] Lustre: Skipped 1 previous similar message [12157.410718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12157.418679] Lustre: Skipped 2 previous similar messages [12161.550163] Lustre: server umount lustre-MDT0000 complete [12163.554522] LustreError: 327290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786113572 with bad export cookie 17017433857348974033 [12163.561091] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12163.563833] LustreError: 327290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12173.874772] Lustre: server umount lustre-MDT0001 complete [12179.500847] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12181.166329] Key type lgssc unregistered [12181.334843] LNet: 330360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12181.339372] LNetError: 330360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12181.348544] LNet: Removed LNI 192.168.204.142@tcp [12181.797780] Key type .llcrypt unregistered [12181.799375] Key type ._llcrypt unregistered [12193.902904] Key type ._llcrypt registered [12193.904667] Key type .llcrypt registered [12193.965371] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12194.431000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12194.481885] alg: No test for adler32 (adler32-zlib) [12195.380443] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [12195.498416] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12197.111453] Key type lgssc registered [12197.705840] Lustre: Echo OBD driver; http://www.lustre.org/ [12201.867604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12201.877738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12203.117256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12205.391542] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12209.720392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12209.915751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12211.991953] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12215.589960] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12218.369554] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12221.844776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12222.054978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12225.249115] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12228.829072] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12238.378949] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12239.141370] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12244.963624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12244.965269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12244.972427] Lustre: Skipped 2 previous similar messages [12244.974269] Lustre: Skipped 1 previous similar message [12245.811231] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [12245.814947] Lustre: Skipped 1 previous similar message [12247.298750] Lustre: server umount lustre-MDT0000 complete [12249.380881] LustreError: 331593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786113658 with bad export cookie 5802182109247426880 [12249.390343] LustreError: 331593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12249.391656] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12250.081459] LustreError: 331606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12250.082607] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12250.096241] LustreError: 331606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12250.101417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12250.914213] LustreError: 331607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12254.112316] LustreError: 334398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12254.113945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12254.131285] Lustre: Skipped 1 previous similar message [12255.670365] Lustre: server umount lustre-MDT0001 complete [12263.854724] Lustre: server umount lustre-OST0000 complete [12268.131204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12270.284237] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12271.705123] Lustre: 335788:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.142@tcp [12272.749865] Lustre: server umount MGS complete [12277.029657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12277.280920] LustreError: 336359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12277.331393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12279.408322] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12282.338486] LustreError: 336360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12283.759389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12283.973574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12286.300825] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12291.183947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12294.351207] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12298.180778] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12299.040493] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12301.691236] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12307.421433] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [12313.058685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12313.065244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12313.070536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12313.075330] Lustre: Skipped 1 previous similar message [12318.311244] Lustre: server umount lustre-OST0000 complete [12323.298161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12323.305893] Lustre: Skipped 1 previous similar message [12323.309387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12323.313320] Lustre: Skipped 3 previous similar messages [12326.204084] Lustre: server umount lustre-MDT0000 complete [12327.940369] LustreError: 339042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786113736 with bad export cookie 5802182109247427720 [12327.942030] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12327.947471] LustreError: 339042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12328.099780] Lustre: server umount lustre-MDT0001 complete [12332.906933] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12334.219186] Key type lgssc unregistered [12334.382882] LNet: 339796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12334.387585] LNetError: 339796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12334.397152] LNet: Removed LNI 192.168.204.142@tcp [12334.810956] Key type .llcrypt unregistered [12334.812757] Key type ._llcrypt unregistered [12343.153955] Key type ._llcrypt registered [12343.155564] Key type .llcrypt registered [12343.208314] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12351.287958] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12351.646048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12351.668949] alg: No test for adler32 (adler32-zlib) [12352.585012] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [12352.707525] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12354.303096] Key type lgssc registered [12354.823770] Lustre: Echo OBD driver; http://www.lustre.org/ [12380.600322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12380.609170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12381.812989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12381.839287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12381.908326] Lustre: lustre-MDT0000: new disk, initializing [12382.014813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12382.039484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12385.752079] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12396.229374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12396.343668] Lustre: 342931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12396.386524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12396.389588] Lustre: Skipped 1 previous similar message [12396.450907] Lustre: lustre-MDT0001: new disk, initializing [12396.533434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12396.573122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12396.587459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12402.043698] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12411.514373] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12417.495337] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12425.277136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12425.572387] Lustre: lustre-OST0000: new disk, initializing [12425.575599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12425.582137] Lustre: 344902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12425.677319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12432.241995] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12432.521675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12432.548196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12432.735866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12440.770810] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12444.232243] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12444.422838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12448.122489] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12448.324720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12453.349852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.350285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12453.363039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12453.365797] LustreError: Skipped 1 previous similar message [12456.123876] Lustre: server umount lustre-OST0000 complete [12463.072704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12463.078431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12463.085486] Lustre: Skipped 1 previous similar message [12463.089539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12463.094125] Lustre: Skipped 1 previous similar message [12465.388697] Lustre: server umount lustre-MDT0000 complete [12468.033649] LustreError: 344901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786113876 with bad export cookie 3616018435625636146 [12468.035420] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12468.042090] LustreError: 344901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12468.280556] Lustre: server umount lustre-MDT0001 complete [12474.158754] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 10:44:41 (1786113881) [12483.930137] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12491.170581] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12516.656760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12516.837953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12516.858024] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12516.935956] Lustre: lustre-MDT0000: new disk, initializing [12516.997805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12517.011579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12520.125804] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12527.808954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12527.956190] Lustre: 349653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12528.016240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12528.020477] Lustre: Skipped 1 previous similar message [12528.112212] Lustre: lustre-MDT0001: new disk, initializing [12528.185914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12528.216889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12528.227284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12531.201286] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12537.847646] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12541.449383] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12546.423382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12546.652959] Lustre: lustre-OST0000: new disk, initializing [12546.674218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12546.678890] Lustre: 351621:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12546.738558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12550.993583] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12554.776890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12554.785975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12554.851733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12557.689168] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12580.324041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12580.325208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12580.329393] LustreError: Skipped 1 previous similar message [12580.330439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12580.338951] Lustre: Skipped 2 previous similar messages [12580.343993] Lustre: Skipped 1 previous similar message [12584.154214] Lustre: server umount lustre-OST0000 complete [12589.536768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12589.541058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12589.552880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12595.682326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12595.692178] Lustre: Skipped 3 previous similar messages [12601.311961] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12601.462776] Lustre: server umount lustre-MDT0000 complete [12604.215514] LustreError: 349647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786114012 with bad export cookie 3616018435625637028 [12604.222119] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12604.229110] LustreError: 349647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12604.457553] Lustre: server umount lustre-MDT0001 complete [12612.481669] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12614.435546] Key type lgssc unregistered [12614.672101] LNet: 353777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12614.678815] LNetError: 353777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12614.692403] LNet: Removed LNI 192.168.204.142@tcp [12615.328995] Key type .llcrypt unregistered [12615.331601] Key type ._llcrypt unregistered [12678.154056] Key type ._llcrypt registered [12678.155711] Key type .llcrypt registered [12678.237287] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12678.507402] Key type .llcrypt unregistered [12678.512125] Key type ._llcrypt unregistered [12727.646035] Key type ._llcrypt registered [12727.649397] Key type .llcrypt registered [12727.731833] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12727.932691] Key type .llcrypt unregistered [12727.934824] Key type ._llcrypt unregistered [12747.178619] Key type ._llcrypt registered [12747.180259] Key type .llcrypt registered [12747.251938] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12747.535487] Key type .llcrypt unregistered [12747.538172] Key type ._llcrypt unregistered [12768.625617] Key type ._llcrypt registered [12768.627529] Key type .llcrypt registered [12768.716482] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12769.008527] Key type .llcrypt unregistered [12769.010816] Key type ._llcrypt unregistered [12821.099743] Key type ._llcrypt registered [12821.101744] Key type .llcrypt registered [12821.178983] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12821.387757] Key type .llcrypt unregistered [12821.390447] Key type ._llcrypt unregistered [12843.097355] Key type ._llcrypt registered [12843.106178] Key type .llcrypt registered [12843.226989] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12843.523723] Key type .llcrypt unregistered [12843.526172] Key type ._llcrypt unregistered [12865.039394] Key type ._llcrypt registered [12865.040981] Key type .llcrypt registered [12865.135994] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12865.374365] Key type .llcrypt unregistered [12865.377493] Key type ._llcrypt unregistered [12885.243951] Key type ._llcrypt registered [12885.245619] Key type .llcrypt registered [12885.325815] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12885.552326] Key type .llcrypt unregistered [12885.555239] Key type ._llcrypt unregistered [12904.593290] Key type ._llcrypt registered [12904.595116] Key type .llcrypt registered [12904.668225] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12904.863325] Key type .llcrypt unregistered [12904.865762] Key type ._llcrypt unregistered [12923.800429] Key type ._llcrypt registered [12923.802480] Key type .llcrypt registered [12923.887368] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12924.115225] Key type .llcrypt unregistered [12924.117953] Key type ._llcrypt unregistered [12961.857121] Key type ._llcrypt registered [12961.859037] Key type .llcrypt registered [12961.950923] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12962.200861] Key type .llcrypt unregistered [12962.203520] Key type ._llcrypt unregistered [12981.672838] Key type ._llcrypt registered [12981.674226] Key type .llcrypt registered [12981.725709] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12981.895542] Key type .llcrypt unregistered [12981.897395] Key type ._llcrypt unregistered [13001.142185] Key type ._llcrypt registered [13001.143880] Key type .llcrypt registered [13001.237704] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13001.525968] Key type .llcrypt unregistered [13001.528208] Key type ._llcrypt unregistered [13033.904971] Key type ._llcrypt registered [13033.906169] Key type .llcrypt registered [13033.986243] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13044.519714] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13045.163648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13045.196918] alg: No test for adler32 (adler32-zlib) [13046.204916] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [13046.370361] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [13048.015259] Key type lgssc registered [13048.815446] Lustre: Echo OBD driver; http://www.lustre.org/ [13069.815694] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13069.835403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13071.008761] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13071.037523] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13071.086623] Lustre: tst250fs-MDT0000: new disk, initializing [13071.133518] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13071.145565] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13073.944149] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13080.973719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13081.045500] Lustre: 373762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13081.092649] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13081.097380] Lustre: Skipped 1 previous similar message [13081.192276] Lustre: tst250fs-MDT0001: new disk, initializing [13081.275206] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13081.312379] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13081.337926] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13084.012156] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13087.326791] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13091.365907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13091.548601] Lustre: tst250fs-OST0000: new disk, initializing [13091.552281] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13091.558834] Lustre: 375391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13091.644113] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13093.466220] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13093.472410] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13093.543985] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13095.102403] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13101.653807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13101.806861] Lustre: tst250fs-OST0001: new disk, initializing [13101.810220] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13101.815218] Lustre: 376263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13101.880823] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13105.620794] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13109.575903] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13110.477985] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13110.485288] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13110.594442] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13126.117874] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13126.119089] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13126.125621] Lustre: Skipped 1 previous similar message [13126.136883] Lustre: Skipped 3 previous similar messages [13130.409445] Lustre: server umount tst250fs-MDT0000 complete [13132.942340] LustreError: 374556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786114541 with bad export cookie 4184970270724182769 [13132.945404] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13132.947706] LustreError: 374556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13133.149422] Lustre: server umount tst250fs-MDT0001 complete [13145.638737] Lustre: server umount tst250fs-OST0000 complete [13157.537362] Lustre: server umount tst250fs-OST0001 complete [13170.156145] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13176.316891] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13196.385094] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 10:56:43 (1786114603) [13197.606711] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13207.932186] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13214.186328] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13234.225871] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12871 sec ========= 10:57:21 (1786114641) [13235.276114] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:57:22 (1786114642) === [13237.683890] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:57:24 (1786114644) === [13261.763354] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13263.145473] Key type lgssc unregistered [13263.298555] LNet: 385403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13263.303827] LNetError: 385403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13263.312509] LNet: Removed LNI 192.168.204.142@tcp [13263.750172] Key type .llcrypt unregistered [13263.752228] Key type ._llcrypt unregistered