[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475808769 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002357] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007410] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010126] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.014007] *** VALIDATE selinux *** [ 0.022046] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026819] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] *** VALIDATE tmpfs *** [ 0.031283] *** VALIDATE proc *** [ 0.032247] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.039005] Speculative Store Bypass: Vulnerable [ 0.042170] debug: unmapping init [mem 0xffffffffb6659000-0xffffffffb6660fff] [ 0.045196] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046695] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054331] rcu: Hierarchical SRCU implementation. [ 0.056431] smp: Bringing up secondary CPUs ... [ 0.057589] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.061692] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.140030] node 0 deferred pages initialised in 73ms [ 0.146024] devtmpfs: initialized [ 0.147228] x86/mm: Memory block size: 128MB [ 0.149720] gcov: version magic: 0x41383552 [ 0.153368] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160314] pinctrl core: initialized pinctrl subsystem [ 0.162169] [ 0.162763] ************************************************************* [ 0.165015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168014] ** ** [ 0.171012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174013] ** ** [ 0.177015] ** This means that this kernel is built to expose internal ** [ 0.179013] ** IOMMU data structures, which may compromise security on ** [ 0.182014] ** your system. ** [ 0.184012] ** ** [ 0.187013] ** If you see this message and you are not debugging the ** [ 0.190014] ** kernel, report this immediately to your vendor! ** [ 0.192011] ** ** [ 0.195013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198014] ************************************************************* [ 0.200662] NET: Registered protocol family 16 [ 0.202411] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212112] cpuidle: using governor menu [ 0.215081] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217427] PCI: Using configuration type 1 for base access [ 0.219118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.229295] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.232098] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.235074] cryptd: max_cpu_qlen set to 1000 [ 0.237214] ACPI: Added _OSI(Module Device) [ 0.239013] ACPI: Added _OSI(Processor Device) [ 0.240030] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.242019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248432] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255466] ACPI: Interpreter enabled [ 0.257077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.259012] ACPI: Using IOAPIC for interrupt routing [ 0.261092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.265446] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276948] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.284024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.295290] acpiphp: Slot [2] registered [ 0.297100] acpiphp: Slot [5] registered [ 0.298115] acpiphp: Slot [6] registered [ 0.300143] acpiphp: Slot [7] registered [ 0.301127] acpiphp: Slot [8] registered [ 0.303133] acpiphp: Slot [9] registered [ 0.305118] acpiphp: Slot [10] registered [ 0.306120] acpiphp: Slot [3] registered [ 0.308118] acpiphp: Slot [4] registered [ 0.310128] acpiphp: Slot [11] registered [ 0.312132] acpiphp: Slot [12] registered [ 0.313120] acpiphp: Slot [13] registered [ 0.315119] acpiphp: Slot [14] registered [ 0.317113] acpiphp: Slot [15] registered [ 0.319111] acpiphp: Slot [16] registered [ 0.320102] acpiphp: Slot [17] registered [ 0.322155] acpiphp: Slot [18] registered [ 0.324108] acpiphp: Slot [19] registered [ 0.325108] acpiphp: Slot [20] registered [ 0.327094] acpiphp: Slot [21] registered [ 0.329100] acpiphp: Slot [22] registered [ 0.330104] acpiphp: Slot [23] registered [ 0.332101] acpiphp: Slot [24] registered [ 0.333089] acpiphp: Slot [25] registered [ 0.335094] acpiphp: Slot [26] registered [ 0.336111] acpiphp: Slot [27] registered [ 0.338101] acpiphp: Slot [28] registered [ 0.339071] acpiphp: Slot [29] registered [ 0.340088] acpiphp: Slot [30] registered [ 0.341070] acpiphp: Slot [31] registered [ 0.342043] PCI host bridge to bus 0000:00 [ 0.343014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.346020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.348023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.354026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.356025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.359056] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.363058] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365468] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.375772] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.381058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.383015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.385016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.387017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.389585] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.394076] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.396044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.399761] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.405010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.417017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.423021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.430748] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.460015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.483016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.508015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.523802] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.532015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.540017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.560017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.571000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.579017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.591016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.611030] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.622118] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.630016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.636013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.654018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.664204] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.670017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.679018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.696014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.709687] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.715019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.721016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.741026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.753000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.756336] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.758330] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.760851] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.763302] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.768180] iommu: Default domain type: Passthrough [ 0.770595] SCSI subsystem initialized [ 0.773249] ACPI: bus type USB registered [ 0.775194] usbcore: registered new interface driver usbfs [ 0.777090] usbcore: registered new interface driver hub [ 0.779085] usbcore: registered new device driver usb [ 0.781205] pps_core: LinuxPPS API ver. 1 registered [ 0.783014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.787062] PTP clock support registered [ 0.789182] EDAC MC: Ver: 3.0.0 [ 0.792122] PCI: Using ACPI for IRQ routing [ 0.794052] NetLabel: Initializing [ 0.796013] NetLabel: domain hash size = 128 [ 0.798024] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.799064] NetLabel: unlabeled traffic allowed by default [ 0.801124] vgaarb: loaded [ 0.803389] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.806025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.812018] clocksource: Switched to clocksource kvm-clock [ 0.929163] VFS: Disk quotas dquot_6.6.0 [ 0.931057] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.933995] *** VALIDATE ramfs *** [ 0.935380] *** VALIDATE hugetlbfs *** [ 0.937023] pnp: PnP ACPI init [ 0.939357] pnp: PnP ACPI: found 6 devices [ 0.962455] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.966941] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.969695] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.972900] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.976032] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.978942] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.983854] NET: Registered protocol family 2 [ 0.987412] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.993080] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.996784] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.004215] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.008954] TCP: Hash tables configured (established 65536 bind 65536) [ 1.013249] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.017880] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.022001] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.025460] NET: Registered protocol family 1 [ 1.029870] RPC: Registered named UNIX socket transport module. [ 1.032614] RPC: Registered udp transport module. [ 1.034270] RPC: Registered tcp transport module. [ 1.036406] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.038714] NET: Registered protocol family 44 [ 1.040447] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.042551] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.044689] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.047294] PCI: CLS 0 bytes, default 64 [ 1.049179] Unpacking initramfs... [ 2.519707] debug: unmapping init [mem 0xffff9450fcc54000-0xffff9450fffbffff] [ 2.523925] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.526797] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.530573] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.051924] Initialise system trusted keyrings [ 3.055463] Key type blacklist registered [ 3.058135] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.070356] zbud: loaded [ 3.076917] *** VALIDATE nfs *** [ 3.078285] *** VALIDATE nfs4 *** [ 3.081396] pstore: using deflate compression [ 3.085859] Platform Keyring initialized [ 3.206748] NET: Registered protocol family 38 [ 3.210931] Key type asymmetric registered [ 3.214862] Asymmetric key parser 'x509' registered [ 3.219269] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.226329] io scheduler mq-deadline registered [ 3.229031] io scheduler kyber registered [ 3.230621] io scheduler bfq registered [ 3.232985] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.236724] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.239925] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.243339] ACPI: Power Button [PWRF] [ 3.251888] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.263729] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.290225] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.298029] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.314415] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.344768] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.375654] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.381097] Non-volatile memory driver v1.3 [ 3.383204] Linux agpgart interface v0.103 [ 3.416132] virtio_blk virtio1: [vda] 150672 512-byte logical blocks (77.1 MB/73.6 MiB) [ 3.419547] vda: detected capacity change from 0 to 77144064 [ 3.433966] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.436392] vdb: detected capacity change from 0 to 1073741824 [ 3.455820] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.460896] vdc: detected capacity change from 0 to 2621440000 [ 3.476154] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.479493] vdd: detected capacity change from 0 to 2621440000 [ 3.493920] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.496801] vde: detected capacity change from 0 to 4294967296 [ 3.509539] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.513089] vdf: detected capacity change from 0 to 4294967296 [ 3.520951] libphy: Fixed MDIO Bus: probed [ 3.530352] usbcore: registered new interface driver usbserial_generic [ 3.532622] usbserial: USB Serial support registered for generic [ 3.535167] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.538976] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.540981] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.542929] mousedev: PS/2 mouse device common for all mice [ 3.546264] rtc_cmos 00:05: RTC can wake from S4 [ 3.548841] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.550854] rtc_cmos 00:05: registered as rtc0 [ 3.555949] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.559192] intel_pstate: CPU model not supported [ 3.560576] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.564951] hid: raw HID events driver (C) Jiri Kosina [ 3.565096] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.566892] usbcore: registered new interface driver usbhid [ 3.571661] usbhid: USB HID core driver [ 3.572900] drop_monitor: Initializing network drop monitor service [ 3.574766] Initializing XFRM netlink socket [ 3.576449] NET: Registered protocol family 10 [ 3.579590] Segment Routing with IPv6 [ 3.581080] NET: Registered protocol family 17 [ 3.583664] mpls_gso: MPLS GSO support [ 3.590117] RAS: Correctable Errors collector initialized. [ 3.592640] AVX version of gcm_enc/dec engaged. [ 3.594480] AES CTR mode by8 optimization enabled [ 3.675204] sched_clock: Marking stable (3675171909, 0)->(4615601340, -940429431) [ 3.679274] registered taskstats version 1 [ 3.681478] Loading compiled-in X.509 certificates [ 3.683585] zswap: loaded using pool lzo/zbud [ 3.710189] Key type big_key registered [ 3.725069] Key type encrypted registered [ 3.726936] ima: No TPM chip found, activating TPM-bypass! [ 3.729334] ima: Allocated hash algorithm: sha1 [ 3.731259] ima: No architecture policies found [ 3.733240] evm: Initialising EVM extended attributes: [ 3.735421] evm: security.selinux [ 3.736922] evm: security.ima [ 3.738171] evm: security.capability [ 3.739542] evm: HMAC attrs: 0x1 [ 3.744780] rtc_cmos 00:05: setting system clock to 2026-09-30 03:35:08 UTC (1790739308) [ 3.751838] debug: unmapping init [mem 0xffffffffb7603000-0xffffffffb77fffff] [ 3.755312] debug: unmapping init [mem 0xffffffffb6382000-0xffffffffb6658fff] [ 3.763354] Write protecting the kernel read-only data: 28672k [ 3.767299] debug: unmapping init [mem 0xffffffffb4a03000-0xffffffffb4bfffff] [ 3.770649] debug: unmapping init [mem 0xffffffffb5314000-0xffffffffb53fffff] [ 3.819089] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.827760] systemd[1]: Detected virtualization kvm. [ 3.829072] systemd[1]: Detected architecture x86-64. [ 3.832075] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.861067] systemd[1]: No hostname configured. [ 3.863138] systemd[1]: Set hostname to . [ 3.865232] random: systemd: uninitialized urandom read (16 bytes read) [ 3.868305] systemd[1]: Initializing machine ID from random generator. [ 3.930843] random: ln: uninitialized urandom read (6 bytes read) [ 4.039162] random: systemd: uninitialized urandom read (16 bytes read) [ 4.044600] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.050537] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.061125] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.701324] device-mapper: uevent: version 1.0.3 [ 4.703824] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.482436] virtio_net virtio0 ens2: renamed from eth0 [ 5.490636] random: fast init done [ 5.563192] scsi host0: ata_piix [ 5.594259] scsi host1: ata_piix [ 5.596315] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.599187] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.000340] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.247206] random: crng init done [ 10.248123] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.765836] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.962483] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.234168] SELinux: Disabled at runtime. [ 12.302498] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.312648] systemd[1]: Detected virtualization kvm. [ 12.314808] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.842382] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.846134] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.854739] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.859614] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.863201] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.886623] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.892916] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [[ 13.006249] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.332255] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.682807] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.733645] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.854301] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.866659] EDAC sbridge: Ver: 1.1.2 [ 15.693590] Key type dns_resolver registered [ 15.997422] NFS: Registering the id_resolver key type [ 15.999744] Key type id_resolver registered [ 16.002010] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg331-server login: [ 60.913819] libcfs: loading out-of-tree module taints kernel. [ 61.023076] Key type ._llcrypt registered [ 61.024785] Key type .llcrypt registered [ 61.185443] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 81.032213] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 82.531130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.555847] alg: No test for adler32 (adler32-zlib) [ 84.124036] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 85.028389] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 85.225941] hrtimer: interrupt took 7747142 ns [ 87.047267] Key type lgssc registered [ 88.854484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 109.708862] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.671129] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 170.317286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 170.369989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.785682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.879294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.065889] Lustre: lustre-MDT0000: new disk, initializing [ 172.176787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.217613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.172780] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.840303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.963566] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 192.990344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 192.998459] Lustre: Skipped 1 previous similar message [ 193.079778] Lustre: lustre-MDT0001: new disk, initializing [ 193.155073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 193.209949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 193.222427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 198.410261] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.342817] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 213.620103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.909797] Lustre: lustre-OST0000: new disk, initializing [ 213.915667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 213.923875] Lustre: 8445:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.993904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.655511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.665779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.715243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.318256] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.786976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.907600] Lustre: lustre-OST0001: new disk, initializing [ 234.912412] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 234.919303] Lustre: 9518:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.976893] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.596875] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.236489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 243.246212] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 243.345071] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 253.739748] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 261.186576] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 269.395417] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing check_logdir /tmp/testlogs/ [ 275.483094] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing yml_node [ 279.700563] Lustre: DEBUG MARKER: Client: 2.17.59.3 [ 282.616869] Lustre: DEBUG MARKER: MDS: 2.17.59.3 [ 285.449482] Lustre: DEBUG MARKER: OSS: 2.17.59.3 [ 287.278982] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 29 23:39:49 EDT 2026 [ 307.363628] Lustre: DEBUG MARKER: only running tests: 121-999 [ 309.247427] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 311.108448] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.211673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.224917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.238813] Lustre: Skipped 3 previous similar messages [ 330.245943] Lustre: Skipped 2 previous similar messages [ 334.143662] Lustre: server umount lustre-MDT0000 complete [ 338.011912] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739642 with bad export cookie 4049240387262286702 [ 338.014921] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.030850] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 340.451759] LustreError: 8453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.451965] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.455272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.455344] Lustre: Skipped 1 previous similar message [ 340.476119] LustreError: 8453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 340.508024] Lustre: Skipped 1 previous similar message [ 343.455906] LustreError: 8453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.455969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.504042] Lustre: Skipped 2 previous similar messages [ 344.490747] Lustre: server umount lustre-MDT0001 complete [ 348.538268] Lustre: server umount lustre-OST0000 complete [ 352.816485] Lustre: server umount lustre-OST0001 complete [ 360.160373] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 370.496870] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 408.893521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.215776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 409.238256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 409.361597] Lustre: lustre-MDT0000: new disk, initializing [ 409.505901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.544453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 413.931290] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 423.564052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.702316] Lustre: 15121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 423.803664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 423.811428] Lustre: Skipped 1 previous similar message [ 423.930092] Lustre: lustre-MDT0001: new disk, initializing [ 424.030937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 424.079401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 424.094504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 428.542898] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 438.442929] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 444.678951] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 452.086305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 452.541874] Lustre: lustre-OST0000: new disk, initializing [ 452.550281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 452.560973] Lustre: 17090:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 452.672499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 454.012328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 454.026446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 454.172488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 458.834721] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 468.494669] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 473.200838] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 473.494284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 478.979252] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 479.324140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 484.832679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 484.836326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.843363] LustreError: Skipped 1 previous similar message [ 484.846455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 484.854772] Lustre: Skipped 1 previous similar message [ 487.686774] Lustre: server umount lustre-OST0000 complete [ 495.073117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.098906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.107162] Lustre: Skipped 2 previous similar messages [ 504.297481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 504.308621] Lustre: Skipped 3 previous similar messages [ 506.335673] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 506.598236] Lustre: server umount lustre-MDT0000 complete [ 509.409519] LustreError: 15125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.433899] LustreError: 15125:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 510.677866] LustreError: 15916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739815 with bad export cookie 4049240387262288235 [ 510.690503] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.695806] LustreError: 15916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 511.189675] Lustre: server umount lustre-MDT0001 complete [ 516.788421] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:43:39 (1790739819) [ 537.755390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.246799] LustreError: 19967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.339533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.714972] LustreError: 19968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.720708] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 553.957429] LustreError: 19967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.986811] LustreError: 19967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 555.008090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.602681] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 569.076516] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 575.963278] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 578.804703] Lustre: Failing over lustre-MDT0000 [ 578.994240] Lustre: server umount lustre-MDT0000 complete [ 580.897173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.904853] LustreError: 19967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.923207] Lustre: Skipped 2 previous similar messages [ 580.971138] LustreError: 19967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 596.450862] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790739885/real 1790739885] req@ffff9450469c5c00 x1877726345724416/t0(0) o400->MGC192.168.203.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790739901 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 596.492268] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.250454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.861861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.869336] Lustre: Skipped 1 previous similar message [ 607.917485] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 612.167563] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 613.348665] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 613.363698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 613.388953] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 622.579501] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 628.705519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 628.717346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.736782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.278173] Lustre: server umount lustre-MDT0000 complete [ 636.212471] LustreError: 19947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739940 with bad export cookie 4049240387262289649 [ 636.223020] LustreError: 19947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 636.228323] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.596214] Lustre: server umount lustre-MDT0001 complete [ 644.090869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.579443] LustreError: 23306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.604342] LustreError: 23306:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 649.297421] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 657.179725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.669441] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 668.868375] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 673.998510] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 675.915174] Lustre: Failing over lustre-MDT0000 [ 676.093638] Lustre: server umount lustre-MDT0000 complete [ 677.856988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 677.875930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.898034] Lustre: Skipped 1 previous similar message [ 696.235398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.346715] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 696.617717] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 700.878369] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 701.921402] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 701.936631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 701.942734] Lustre: Skipped 1 previous similar message [ 701.966747] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 710.566646] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 717.282500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.290899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 717.299918] Lustre: Skipped 2 previous similar messages [ 717.321198] Lustre: Skipped 3 previous similar messages [ 720.868099] Lustre: server umount lustre-MDT0000 complete [ 722.400976] LustreError: 24034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.428694] LustreError: 24034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 724.838382] LustreError: 23287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740029 with bad export cookie 4049240387262290587 [ 724.845647] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 724.847240] LustreError: 23287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 725.261923] Lustre: server umount lustre-MDT0001 complete [ 735.681947] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:47:18 (1790740038) [ 752.222111] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 761.630356] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 804.399204] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 816.516222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.784530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 816.807552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 816.904516] Lustre: lustre-MDT0000: new disk, initializing [ 817.005635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 817.009705] Lustre: Skipped 3 previous similar messages [ 817.024682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 822.208953] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 834.106548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.250649] Lustre: 29678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 834.673841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 839.310352] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 849.957044] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 858.124747] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 866.135295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 866.637719] Lustre: 31647:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 867.790295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 867.845247] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 867.911672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 867.933815] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 874.424875] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 884.661569] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 890.049284] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 913.892434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 913.896687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.964668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 913.994092] Lustre: Skipped 1 previous similar message [ 919.135943] Lustre: server umount lustre-OST0000 complete [ 928.916698] Lustre: server umount lustre-MDT0000 complete [ 929.252909] LustreError: 29690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.272601] LustreError: 29690:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 932.671423] LustreError: 29670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740237 with bad export cookie 4049240387262290986 [ 932.692846] LustreError: 29670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 932.693547] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 933.288700] Lustre: server umount lustre-MDT0001 complete [ 943.837248] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 946.750843] Key type lgssc unregistered [ 947.035277] LNet: 33630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.041842] LNetError: 33630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.056869] LNet: Removed LNI 192.168.203.131@tcp [ 947.862888] Key type .llcrypt unregistered [ 947.865145] Key type ._llcrypt unregistered [ 971.193590] Key type ._llcrypt registered [ 971.198626] Key type .llcrypt registered [ 971.320503] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 972.325837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 972.409943] alg: No test for adler32 (adler32-zlib) [ 973.514610] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 973.856362] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 975.551197] Key type lgssc registered [ 976.735683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 985.526915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 985.549231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.003597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.699208] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 996.321958] LustreError: 35059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.338453] LustreError: 35059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1001.447966] LustreError: 35060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.427317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.920245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1007.785082] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1015.022707] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1020.998852] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1028.752066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1029.457928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.623503] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1047.925827] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1049.234439] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1063.879409] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:52:46 (1790740366) [ 1065.789718] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1067.603972] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1074.926740] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:52:57 (1790740377) [ 1084.429563] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:53:06 (1790740386) [ 1085.439782] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1094.332967] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1094.337584] Lustre: Skipped 2 previous similar messages [ 1095.946913] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:53:18 (1790740398) [ 1099.726149] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1099.732812] Lustre: Skipped 3 previous similar messages [ 1118.042550] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1118.047258] Lustre: Skipped 6 previous similar messages [ 1120.870740] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:53:43 (1790740423) [ 1123.290976] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1126.810388] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1135.653569] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:53:58 (1790740438) [ 1137.306779] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1137.313257] Lustre: Skipped 3 previous similar messages [ 1148.405732] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:54:11 (1790740451) [ 1164.768292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1164.790845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.829153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.797577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.798097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.798266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1170.273739] Lustre: server umount lustre-OST0000 complete [ 1176.034170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.065321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1176.068562] Lustre: Skipped 1 previous similar message [ 1181.153912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1181.161627] Lustre: Skipped 1 previous similar message [ 1185.249373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1185.260211] Lustre: Skipped 1 previous similar message [ 1189.343984] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1189.585139] Lustre: server umount lustre-MDT0000 complete [ 1190.376031] LustreError: 35059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.402763] LustreError: 35059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1193.516806] LustreError: 38978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740498 with bad export cookie 14895682536620096022 [ 1193.521580] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1193.538475] LustreError: 38978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1194.037279] Lustre: server umount lustre-MDT0001 complete [ 1206.219571] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 1209.137854] Key type lgssc unregistered [ 1209.576371] LNet: 42917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.590273] LNetError: 42917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.616814] LNet: Removed LNI 192.168.203.131@tcp [ 1210.445920] Key type .llcrypt unregistered [ 1210.448537] Key type ._llcrypt unregistered [ 1228.868846] Key type ._llcrypt registered [ 1228.871440] Key type .llcrypt registered [ 1229.015830] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 1246.285567] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 1247.137642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1247.185219] alg: No test for adler32 (adler32-zlib) [ 1248.642424] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 1248.982671] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 1250.711222] Key type lgssc registered [ 1252.148370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1292.257116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1292.276033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.604475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1293.672737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1293.768564] Lustre: lustre-MDT0000: new disk, initializing [ 1293.880155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.908210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1298.767578] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1311.338508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.460845] Lustre: 46101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1311.596675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1311.601361] Lustre: Skipped 1 previous similar message [ 1311.759606] Lustre: lustre-MDT0001: new disk, initializing [ 1311.866536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1311.930067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1311.947502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1316.676239] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1326.832949] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1333.823374] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1343.577586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1344.197171] Lustre: lustre-OST0000: new disk, initializing [ 1344.209764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1344.224923] Lustre: 48072:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1344.447661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.773309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1352.790861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1352.882647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1353.166507] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1365.114884] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1371.053562] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1371.349404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1376.747867] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1376.969512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1383.908555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1383.914861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.920028] LustreError: Skipped 1 previous similar message [ 1383.922975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1383.961934] Lustre: Skipped 1 previous similar message [ 1385.501488] Lustre: server umount lustre-OST0000 complete [ 1393.633402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1393.649488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.677990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.681302] Lustre: Skipped 1 previous similar message [ 1399.265112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1399.273438] Lustre: Skipped 3 previous similar messages [ 1403.365488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1403.375519] Lustre: Skipped 1 previous similar message [ 1404.383130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1404.645757] Lustre: server umount lustre-MDT0000 complete [ 1408.449263] LustreError: 46898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740713 with bad export cookie 9041511558496250914 [ 1408.459814] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1408.463226] LustreError: 46898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1408.759710] Lustre: server umount lustre-MDT0001 complete [ 1420.175606] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 23:58:42 (1790740722) [ 1428.204160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.900480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.081173] LustreError: 50534:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.108214] LustreError: 50534:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1434.472461] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1439.209466] LustreError: 50535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.852328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.307336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1449.302291] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1457.525742] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1464.825846] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1473.223841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1473.720339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.528787] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1489.811874] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1494.943875] LustreError: 53062:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1496.136225] Lustre: Modifying parameter general.timeout=1 in log params [ 1497.343366] Lustre: Modifying parameter general.timeout=2 in log params [ 1499.698469] Lustre: Modifying parameter general.timeout=4 in log params [ 1499.710425] Lustre: Skipped 1 previous similar message [ 1504.011171] Lustre: Modifying parameter general.timeout=8 in log params [ 1504.030245] Lustre: Skipped 3 previous similar messages [ 1512.059920] Lustre: Modifying parameter general.timeout=14 in log params [ 1512.066814] Lustre: Skipped 5 previous similar messages [ 1528.668849] Lustre: Modifying parameter general.timeout=28 in log params [ 1528.677095] Lustre: Skipped 13 previous similar messages [ 1561.185957] Lustre: Modifying parameter general.timeout=56 in log params [ 1561.190926] Lustre: Skipped 27 previous similar messages [ 1625.448446] Lustre: Modifying parameter general.timeout=119 in log params [ 1625.452590] Lustre: Skipped 62 previous similar messages [ 1648.547656] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:02:30 (1790740950) [ 1651.567275] Lustre: 60034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1653.702330] Lustre: 60134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1655.794734] Lustre: 60232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1661.724030] Lustre: 60543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1661.740208] Lustre: 60543:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1691.412422] Lustre: 61917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1691.471521] LustreError: 61917:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1696.073300] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:03:18 (1790740998) [ 1710.053281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.055462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1710.074456] Lustre: Skipped 2 previous similar messages [ 1710.091794] Lustre: Skipped 2 previous similar messages [ 1715.438386] Lustre: server umount lustre-MDT0000 complete [ 1719.354754] LustreError: 50515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741024 with bad export cookie 9041511558496251789 [ 1719.369550] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1719.383344] LustreError: 50515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1719.859701] Lustre: server umount lustre-MDT0001 complete [ 1733.625755] Lustre: server umount lustre-OST0000 complete [ 1766.732354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.995751] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1767.035506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1767.043559] Lustre: Skipped 33 previous similar messages [ 1767.347131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.716966] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1782.022343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.187424] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1782.271189] Lustre: 64449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1787.364368] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1795.243644] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1801.313416] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1808.134297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1808.293114] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1808.677160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.681976] Lustre: Skipped 1 previous similar message [ 1816.005085] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1823.873400] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1825.691226] Lustre: 67019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1836.515965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1836.523454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1836.527571] Lustre: Skipped 1 previous similar message [ 1836.529251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1836.529326] Lustre: Skipped 3 previous similar messages [ 1838.699842] Lustre: server umount lustre-OST0000 complete [ 1844.199136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1844.215823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1844.238265] Lustre: Skipped 1 previous similar message [ 1850.457840] Lustre: server umount lustre-MDT0000 complete [ 1851.880181] LustreError: 65389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.913516] LustreError: 65389:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1854.963694] LustreError: 64442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741159 with bad export cookie 9041511558496255016 [ 1854.974412] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1854.990684] LustreError: 64442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1855.285945] Lustre: server umount lustre-MDT0001 complete [ 1867.438201] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 1870.635579] Key type lgssc unregistered [ 1870.942086] LNet: 68230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.961832] LNetError: 68230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.979362] LNet: Removed LNI 192.168.203.131@tcp [ 1871.912522] Key type .llcrypt unregistered [ 1871.920089] Key type ._llcrypt unregistered [ 1899.137170] Key type ._llcrypt registered [ 1899.139840] Key type .llcrypt registered [ 1899.297560] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 1900.319821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1900.347320] alg: No test for adler32 (adler32-zlib) [ 1901.428646] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 1901.723775] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 1903.415182] Key type lgssc registered [ 1904.838042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1913.540972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1913.563732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.209323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1919.852994] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1924.579936] LustreError: 69661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1924.604825] LustreError: 69661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1928.296138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.764656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1932.764470] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1939.873851] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1946.232554] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1953.831135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1954.483434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.250545] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1969.852765] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1978.629355] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1990.112910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1990.141809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.181649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.754645] Lustre: server umount lustre-MDT0000 complete [ 1995.771198] LustreError: 69658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1995.799632] LustreError: 69658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1998.273379] LustreError: 69643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741303 with bad export cookie 15214872554754245074 [ 1998.283572] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1998.295337] LustreError: 69643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1998.738224] Lustre: server umount lustre-MDT0001 complete [ 2012.608980] Lustre: server umount lustre-OST0000 complete [ 2042.716755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.994234] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2043.049757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2043.294668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.440303] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2058.169631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.282976] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2058.310731] Lustre: 74533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2058.913569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2064.071203] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2072.983884] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2079.007500] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2087.187270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2087.524780] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2087.981905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.991881] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2104.133348] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2106.316817] Lustre: 77106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2109.008710] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2121.200557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2121.205914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.234700] Lustre: Skipped 2 previous similar messages [ 2121.255440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2121.268506] Lustre: Skipped 2 previous similar messages [ 2123.141846] Lustre: server umount lustre-OST0000 complete [ 2126.820987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2126.823870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2126.838588] Lustre: Skipped 2 previous similar messages [ 2126.850930] Lustre: Skipped 2 previous similar messages [ 2131.940060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.062707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.081040] Lustre: Skipped 2 previous similar messages [ 2141.151386] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2141.340323] Lustre: server umount lustre-MDT0000 complete [ 2142.193851] LustreError: 74545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.217634] LustreError: 74545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2145.499715] LustreError: 77108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741450 with bad export cookie 15214872554754245900 [ 2145.514627] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2145.526019] LustreError: 77108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2146.039939] Lustre: server umount lustre-MDT0001 complete [ 2157.626537] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 2161.055944] Key type lgssc unregistered [ 2161.347717] LNet: 78327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.359538] LNetError: 78327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.394438] LNet: Removed LNI 192.168.203.131@tcp [ 2162.322490] Key type .llcrypt unregistered [ 2162.324715] Key type ._llcrypt unregistered [ 2186.475675] Key type ._llcrypt registered [ 2186.477951] Key type .llcrypt registered [ 2186.590475] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 2187.781680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2188.043151] alg: No test for adler32 (adler32-zlib) [ 2189.265538] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 2189.564127] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 2191.247199] Key type lgssc registered [ 2192.748590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2203.051000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2203.074202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2204.614601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2208.847988] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2214.880784] LustreError: 79757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.900696] LustreError: 79757:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2217.559505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.029137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2222.154060] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2230.547711] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2237.009535] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2244.120243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2244.595744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.092439] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2259.089628] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2268.776492] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2269.911102] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2271.754937] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2271.768392] Lustre: Skipped 1 previous similar message [ 2274.099694] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2274.103305] Lustre: Skipped 2 previous similar messages [ 2278.419224] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2278.422211] Lustre: Skipped 4 previous similar messages [ 2286.960205] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2286.970125] Lustre: Skipped 9 previous similar messages [ 2303.697788] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2303.704233] Lustre: Skipped 16 previous similar messages [ 2336.006515] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2336.009738] Lustre: Skipped 32 previous similar messages [ 2368.216697] Lustre: 87857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2370.977545] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:14:33 (1790741673) [ 2372.545686] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2374.550422] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:14:37 (1790741677) [ 2378.721785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2378.733093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.745181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2380.769293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2380.773904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.774338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2384.237288] Lustre: server umount lustre-OST0000 complete [ 2391.009072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.020741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2391.027368] Lustre: Skipped 2 previous similar messages [ 2394.482621] Lustre: server umount lustre-MDT0000 complete [ 2396.133420] LustreError: 80784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.155053] LustreError: 80784:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2398.382090] LustreError: 86359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741703 with bad export cookie 16725936536516545089 [ 2398.398719] LustreError: 86359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2398.403974] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2398.845967] Lustre: server umount lustre-MDT0001 complete [ 2411.378360] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 2414.706503] Key type lgssc unregistered [ 2414.978540] LNet: 89107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.992189] LNetError: 89107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2415.006182] LNet: Removed LNI 192.168.203.131@tcp [ 2416.333544] Key type .llcrypt unregistered [ 2416.337330] Key type ._llcrypt unregistered [ 2422.655593] Key type ._llcrypt registered [ 2422.662379] Key type .llcrypt registered [ 2422.782816] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2427.950202] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules [ 2428.653846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2428.712690] alg: No test for adler32 (adler32-zlib) [ 2429.974152] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 2430.264459] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 2432.111330] Key type lgssc registered [ 2434.104912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2435.678920] LustreError: 90338:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2451.273793] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 2457.060160] LustreError: 90338:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2461.120045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2461.145465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.644333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.826797] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2472.934836] LustreError: 91348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.954641] LustreError: 91348:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2474.914506] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:16:17 (1790741777) [ 2478.051842] LustreError: 91349:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2480.347817] LustreError: 91380:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2480.357895] LustreError: 91380:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2480.368346] LustreError: 91380:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2480.775753] Lustre: server umount lustre-MDT0000 complete [ 2496.063694] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 2499.557641] Key type lgssc unregistered [ 2499.989107] LNet: 92872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.998990] LNetError: 92872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2500.020272] LNet: Removed LNI 192.168.203.131@tcp [ 2501.121659] Key type .llcrypt unregistered [ 2501.125179] Key type ._llcrypt unregistered [ 2518.739886] Key type ._llcrypt registered [ 2518.745824] Key type .llcrypt registered [ 2518.861558] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 2519.898828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2520.024642] alg: No test for adler32 (adler32-zlib) [ 2521.111581] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 2521.371097] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 2523.023225] Key type lgssc registered [ 2523.953583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2531.807555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2531.834312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.598929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.605596] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2544.110387] LustreError: 94108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.143115] LustreError: 94108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2547.405460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.022338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2552.170165] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2559.163514] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2564.695826] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2571.985846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2572.827496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.376625] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2587.152258] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2641.836938] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:19:04 (1790741944) [ 2643.880487] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2645.947756] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:19:08 (1790741948) [ 2655.203153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2655.213911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.240993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2655.717907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.742972] Lustre: Skipped 1 previous similar message [ 2655.751762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2655.755851] Lustre: Skipped 1 previous similar message [ 2657.578820] Lustre: server umount lustre-MDT0000 complete [ 2660.839330] LustreError: 94102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.882613] LustreError: 94102:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2661.361599] LustreError: 97286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741966 with bad export cookie 14672962263646848653 [ 2661.366805] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2661.369658] LustreError: 97286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2661.811724] Lustre: server umount lustre-MDT0001 complete [ 2676.392636] Lustre: server umount lustre-OST0000 complete [ 2689.568212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.157977] LustreError: 98404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.388655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.448967] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2695.653319] LustreError: 98405:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.760334] LustreError: 98404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.246531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.078811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2710.631147] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2720.305185] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2726.594717] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2742.125361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2742.426808] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2742.441560] LustreError: 98392:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2742.481332] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2742.507508] LustreError: 100487:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2742.519239] LustreError: 100487:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2742.529041] LustreError: 100487:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2742.639929] Lustre: server umount lustre-OST0000 complete [ 2742.642634] LustreError: 100487:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2751.700356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2751.915058] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2751.925411] LustreError: 98394:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2751.949026] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2751.964500] LustreError: 100954:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2751.972939] LustreError: 100954:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2751.983237] LustreError: 100954:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2752.088360] Lustre: server umount lustre-OST0000 complete [ 2752.092965] LustreError: 100954:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2761.943293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2762.182618] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2762.187093] Lustre: Found index 0 for lustre-OST0000, updating log [ 2762.192994] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2762.285497] Lustre: lustre-OST0000: new disk, initializing [ 2762.295273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2762.305658] Lustre: 101490:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2762.590236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.744137] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2772.839429] Lustre: Failing over lustre-OST0000 [ 2773.082363] Lustre: server umount lustre-OST0000 complete [ 2780.131212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2780.144986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.996712] Lustre: server umount lustre-MDT0000 complete [ 2785.251331] LustreError: 98404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.936207] LustreError: 100971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742091 with bad export cookie 14672962263646851397 [ 2786.943389] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2786.957394] LustreError: 100971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2787.320590] Lustre: server umount lustre-MDT0001 complete [ 2798.110253] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:21:40 (1790742100) [ 2818.527602] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 2829.638607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.122847] LustreError: 103836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.147045] LustreError: 103836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2830.286809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2835.295670] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2844.679130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.799371] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2854.096566] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2861.685315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2862.450711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.460307] Lustre: Skipped 1 previous similar message [ 2869.189691] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2878.503524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2878.634445] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2878.733203] Lustre: lustre-OST0001: new disk, initializing [ 2878.741615] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2878.761595] Lustre: 106153:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2884.706813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2884.721775] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2884.862940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2885.407936] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2893.123282] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2899.437644] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2905.578382] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2905.589743] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2905.608632] Lustre: Skipped 3 previous similar messages [ 2905.626757] Lustre: Skipped 3 previous similar messages [ 2910.697628] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2910.708307] Lustre: Skipped 3 previous similar messages [ 2910.867110] Lustre: server umount lustre-MDT0001 complete [ 2913.352281] LustreError: 106635:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2913.377214] LustreError: 106635:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2920.938518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.092760] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2921.108755] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2921.229718] Lustre: 103823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2922.359400] LustreError: 107685:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2922.365799] LustreError: 107685:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2922.373070] LustreError: 103828:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2922.489398] LustreError: 107696:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.131@tcp: cfg command failed: rc = -17 [ 2922.514383] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2922.514383] [ 2927.098461] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2927.755718] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2939.202575] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:24:01 (1790742241) [ 2959.706055] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 2980.867192] Lustre: 109767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3004.898889] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3008.703755] Lustre: 110904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3100.643269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3100.652220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.663141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3104.130909] Lustre: server umount lustre-MDT0000 complete [ 3106.278574] LustreError: 103836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.304344] LustreError: 103836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 3108.651813] LustreError: 103816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742413 with bad export cookie 14672962263646852384 [ 3108.661041] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3108.664765] LustreError: 103816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3109.121432] Lustre: server umount lustre-MDT0001 complete [ 3124.228070] Lustre: server umount lustre-OST0000 complete [ 3127.647203] Lustre: 93406:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790742416/real 1790742416] req@ffff94504c952300 x1877728901302016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790742432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3127.679578] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.700595] Lustre: Skipped 3 previous similar messages [ 3128.111842] Lustre: server umount lustre-OST0001 complete [ 3134.212047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3144.066690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3153.736642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3164.963847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3185.128539] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 3198.197417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.267106] Lustre: lustre-MDT0000: reset Object Index mappings [ 3199.499391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3199.509357] Lustre: Skipped 2 previous similar messages [ 3206.439943] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3215.692354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3215.716958] Lustre: lustre-MDT0001: reset Object Index mappings [ 3221.321533] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3225.321659] Lustre: 116004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3233.756852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3240.772564] LustreError: 116359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.803526] LustreError: 116359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3240.824389] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3241.140437] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3243.938292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3252.054030] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3257.890028] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3257.904992] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3259.601782] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3268.073265] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3271.703900] Lustre: 117879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3330.423957] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:30:32 (1790742632) [ 3359.722223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3359.729446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.755352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.764262] Lustre: Skipped 4 previous similar messages [ 3361.641536] Lustre: server umount lustre-MDT0000 complete [ 3365.277556] LustreError: 117881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742670 with bad export cookie 14672962263646899305 [ 3365.281318] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3365.299197] LustreError: 117881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3371.903862] Lustre: server umount lustre-MDT0001 complete [ 3376.161485] LustreError: 118645:0:(ldlm_resource.c:1227:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0x7d:0x0].0x0 (ffff94505420df00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3382.435800] Lustre: server umount lustre-OST0000 complete [ 3393.071948] Lustre: server umount lustre-OST0001 complete [ 3401.716342] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 3414.064073] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 3455.389665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.698984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3455.745150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3455.854581] Lustre: lustre-MDT0000: new disk, initializing [ 3455.937961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.941286] Lustre: Skipped 3 previous similar messages [ 3455.955995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3461.442879] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3469.458114] Lustre: server umount lustre-MDT0000 complete [ 3480.079557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.562043] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3480.597296] Lustre: 122157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3480.625715] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3486.311982] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3491.481917] Lustre: server umount lustre-MDT0000 complete [ 3501.212987] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:33:23 (1790742803) [ 3503.280921] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3505.421654] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:33:27 (1790742807) [ 3525.320882] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 3538.020671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.967298] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3555.599748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.808371] Lustre: 124213:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3555.920467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3555.928330] Lustre: Skipped 1 previous similar message [ 3556.086724] Lustre: lustre-MDT0001: new disk, initializing [ 3556.218374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3556.233661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3561.780715] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3567.572665] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3576.226612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3576.946940] Lustre: lustre-OST0000: new disk, initializing [ 3576.959284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3576.983980] Lustre: 125790:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3578.939219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3578.947357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3579.161121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3585.645784] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3598.426907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3598.701298] Lustre: lustre-OST0001: new disk, initializing [ 3598.707266] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3598.723040] Lustre: 126663:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3600.611477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3600.626499] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3600.792959] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3606.134871] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3616.456170] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3620.574419] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4024.293357] Lustre: 133078:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4024.308472] Lustre: 133078:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4024.319914] Lustre: 133078:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4024.330794] Lustre: 133078:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4024.341182] Lustre: 133078:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4024.349338] Lustre: 133078:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4337.905944] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:47:19 (1790743639) [ 4355.058785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4355.071309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4355.097420] Lustre: Skipped 5 previous similar messages [ 4355.107897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4355.115780] Lustre: Skipped 8 previous similar messages [ 4360.150553] Lustre: server umount lustre-MDT0000 complete [ 4363.748846] LustreError: 127470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.779620] LustreError: 127470:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 4364.267802] LustreError: 133118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743669 with bad export cookie 14672962263646968836 [ 4364.281563] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4364.284738] LustreError: 133118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4368.867062] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.895953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4368.897510] Lustre: Skipped 4 previous similar messages [ 4368.912163] Lustre: Skipped 5 previous similar messages [ 4371.255901] Lustre: server umount lustre-MDT0001 complete [ 4389.855181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4389.963977] Lustre: server umount lustre-OST0000 complete [ 4401.684680] Lustre: server umount lustre-OST0001 complete [ 4410.711922] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 4423.504522] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 4464.708605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4465.006318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4465.051873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4465.123724] Lustre: lustre-MDT0000: new disk, initializing [ 4465.236061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.249055] Lustre: Skipped 5 previous similar messages [ 4465.302632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4471.025614] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4482.128407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.256396] Lustre: 136153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4482.323308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4482.333104] Lustre: Skipped 1 previous similar message [ 4482.429385] Lustre: lustre-MDT0001: new disk, initializing [ 4482.571397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4482.587469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4487.960922] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4497.583108] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4504.337595] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4512.390907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4512.758928] Lustre: lustre-OST0000: new disk, initializing [ 4512.767955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4512.785383] Lustre: 138124:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4512.947272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4512.964272] Lustre: Skipped 1 previous similar message [ 4514.601427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4514.617469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4514.811897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4520.420339] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4532.754702] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4544.201331] Lustre: lustre-MDD0000: changelog on [ 4547.100593] Lustre: lustre-MDD0001: changelog on [ 6632.789909] Lustre: lustre-MDD0001: changelog off [ 6634.829953] Lustre: 136159:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6635.787373] Lustre: lustre-MDD0000: changelog off [ 6644.705990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6644.719347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6644.736326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6644.746655] Lustre: Skipped 3 previous similar messages [ 6647.277197] Lustre: server umount lustre-OST0000 complete [ 6653.411621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6653.427077] LustreError: Skipped 1 previous similar message [ 6653.433711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6653.454843] Lustre: Skipped 1 previous similar message [ 6653.462219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6659.470838] Lustre: server umount lustre-MDT0000 complete [ 6660.064368] LustreError: 136999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6660.088225] LustreError: 136999:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6662.974982] LustreError: 136144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745967 with bad export cookie 14672962263647790706 [ 6662.978492] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6662.988883] LustreError: 136144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6663.388046] Lustre: server umount lustre-MDT0001 complete [ 6674.082668] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 6676.708383] Key type lgssc unregistered [ 6677.021347] LNet: 147298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6677.040152] LNetError: 147298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6677.066419] LNet: Removed LNI 192.168.203.131@tcp [ 6678.036231] Key type .llcrypt unregistered [ 6678.039208] Key type ._llcrypt unregistered [ 6699.884844] Key type ._llcrypt registered [ 6699.893074] Key type .llcrypt registered [ 6700.077295] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 6714.973257] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 6715.860377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6715.922800] alg: No test for adler32 (adler32-zlib) [ 6716.972367] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 6717.206049] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 6718.911265] Key type lgssc registered [ 6720.145958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6753.841393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6753.875358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.159238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6755.195178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6755.273316] Lustre: lustre-MDT0000: new disk, initializing [ 6755.319895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6755.336589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6759.606740] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6770.008823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6770.195950] Lustre: 150515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6770.285505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6770.292692] Lustre: Skipped 1 previous similar message [ 6770.406696] Lustre: lustre-MDT0001: new disk, initializing [ 6770.554193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6770.603305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6770.624429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6775.061911] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6785.413931] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6791.659156] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6798.179308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6798.465774] Lustre: lustre-OST0000: new disk, initializing [ 6798.473181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6798.484374] Lustre: 152486:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6798.637422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6805.101767] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6806.611204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6806.624504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6806.778251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6815.938533] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6822.991390] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_module obdecho/obdecho [ 6825.204397] LustreError: 153489:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6825.222178] LustreError: 153489:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6826.970894] LustreError: 153585:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6842.337819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6842.348379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.371279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6842.851478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.863125] Lustre: Skipped 1 previous similar message [ 6846.631485] Lustre: server umount lustre-MDT0000 complete [ 6850.194354] LustreError: 150507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746154 with bad export cookie 13591095378807641703 [ 6850.200920] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6850.209649] LustreError: 150507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6850.577223] Lustre: server umount lustre-MDT0001 complete [ 6864.847236] Lustre: server umount lustre-OST0000 complete [ 6886.198729] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 6896.449771] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 6936.212924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.473733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6936.533300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6936.659701] Lustre: lustre-MDT0000: new disk, initializing [ 6936.763655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6936.779907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6942.968212] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6955.663291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6955.818995] Lustre: 157302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6955.901884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6955.906092] Lustre: Skipped 1 previous similar message [ 6956.020877] Lustre: lustre-MDT0001: new disk, initializing [ 6956.147184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6956.189613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6956.214378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6960.477515] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6969.312525] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6975.155795] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6982.395369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6982.598338] Lustre: lustre-OST0000: new disk, initializing [ 6982.602689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6982.610963] Lustre: 159270:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6982.716093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6989.084307] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6991.981852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6992.002655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6992.144671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6999.683635] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7004.207815] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7004.421293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7009.247317] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7009.406461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7012.835655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7012.847737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7012.863767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7012.863774] Lustre: Skipped 2 previous similar messages [ 7012.906028] LustreError: Skipped 1 previous similar message [ 7017.572584] Lustre: server umount lustre-OST0000 complete [ 7022.563211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7022.575636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7022.600502] Lustre: Skipped 1 previous similar message [ 7022.615062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7022.622498] Lustre: Skipped 1 previous similar message [ 7027.705162] Lustre: server umount lustre-MDT0000 complete [ 7028.194558] LustreError: 157313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.220082] LustreError: 157313:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7031.917069] LustreError: 159269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746336 with bad export cookie 13591095378807642662 [ 7031.927605] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7031.931652] LustreError: 159269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7032.316338] Lustre: server umount lustre-MDT0001 complete [ 7036.075934] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:32:18 (1790746338) [ 7037.618527] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7039.785911] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 01:32:22 (1790746342) [ 7058.384266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7058.740146] LustreError: 162245:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7058.813618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7063.303845] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7064.035957] LustreError: 162246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7069.165622] LustreError: 162245:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7071.512503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.017715] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7083.370590] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7088.688841] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7094.932732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7095.271682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7095.281155] Lustre: Skipped 1 previous similar message [ 7101.517922] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7108.571481] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7113.897746] Lustre: *** cfs_fail_loc=718, val=0*** [ 7114.339414] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.31@tcp (stopping) [ 7114.349074] Lustre: Skipped 2 previous similar messages [ 7116.231102] Lustre: server umount lustre-MDT0000 complete [ 7117.797613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7117.811284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7117.824626] Lustre: Skipped 1 previous similar message [ 7117.838904] LustreError: 162973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7119.812436] LustreError: 162227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746424 with bad export cookie 13591095378807643537 [ 7119.814414] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7119.820472] Lustre: *** cfs_fail_loc=718, val=0*** [ 7119.820480] Lustre: Skipped 3 previous similar messages [ 7119.822271] LustreError: 162227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7120.129383] Lustre: server umount lustre-MDT0001 complete [ 7129.416661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.352865] Lustre: *** cfs_fail_loc=723, val=0*** [ 7130.355511] Lustre: Skipped 1 previous similar message [ 7130.463741] LustreError: 165597:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 7130.474554] LustreError: 165597:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.131@tcp: cfg command failed: rc = -122 [ 7130.489898] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7130.489898] [ 7130.503589] LustreError: MGC192.168.203.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7130.529818] LustreError: 165569:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 7130.540753] LustreError: 165569:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -122 [ 7130.560634] LustreError: 165569:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 7130.566870] LustreError: 165569:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7130.683706] Lustre: server umount lustre-MDT0000 complete [ 7130.686686] LustreError: 165569:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 7139.295943] Lustre: 148729:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790746428/real 1790746428] req@0000000096c3e1bb x1877733299828480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790746444 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7139.357046] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.380155] Lustre: Skipped 2 previous similar messages [ 7139.442147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7139.665776] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7139.677545] LustreError: 162237:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 7139.971128] LustreError: 166004:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.998840] LustreError: 166004:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7140.301459] Lustre: 148731:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790746429/real 1790746429] req@00000000288d044e x1877733299828608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790746445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7144.758869] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7145.319263] Lustre: 148730:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790746433/real 1790746433] req@00000000ce996ebb x1877733299828736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790746449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7145.371624] Lustre: 148730:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7150.431926] Lustre: 148729:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790746439/real 1790746439] req@0000000066659d36 x1877733299830400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790746455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7154.528086] Lustre: 148730:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790746443/real 1790746443] req@000000007af2213d x1877733299830784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790746459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7155.364822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.783184] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7158.836618] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7158.840154] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7159.372809] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7159.376963] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7159.383025] Lustre: Skipped 1 previous similar message [ 7159.852658] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7167.079758] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7168.546636] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7174.388168] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7175.894990] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7188.721389] Lustre: server umount lustre-OST0000 complete [ 7194.604058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7194.608165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7194.624145] Lustre: Skipped 1 previous similar message [ 7204.836405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7204.844442] Lustre: Skipped 4 previous similar messages [ 7205.856060] Lustre: 148731:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790746494/real 1790746494] req@00000000fcc71476 x1877733299857536/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790746510 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7206.367163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7206.658646] Lustre: server umount lustre-MDT0000 complete [ 7210.290837] LustreError: 162226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746515 with bad export cookie 13591095378807644468 [ 7210.292492] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7210.302661] LustreError: 162226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7210.619268] Lustre: server umount lustre-MDT0001 complete [ 7221.386494] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 7224.075185] Key type lgssc unregistered [ 7224.337168] LNet: 168824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7224.352791] LNetError: 168824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7224.378135] LNet: Removed LNI 192.168.203.131@tcp [ 7225.214097] Key type .llcrypt unregistered [ 7225.217082] Key type ._llcrypt unregistered [ 7245.577935] Key type ._llcrypt registered [ 7245.580398] Key type .llcrypt registered [ 7245.717527] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 7246.996492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7247.008500] alg: No test for adler32 (adler32-zlib) [ 7248.060384] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 7248.304166] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 7249.975466] Key type lgssc registered [ 7251.181389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7258.488995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7258.507933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7260.049926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7264.981302] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7269.355826] LustreError: 170155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7269.377315] LustreError: 170155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7273.346172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7273.713549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7278.597799] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7285.990451] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7292.078894] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7298.415510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7298.808015] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7304.942198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7306.132599] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7314.205639] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7329.762770] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7329.773342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7329.787045] Lustre: Skipped 1 previous similar message [ 7332.439112] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.31@tcp (stopping) [ 7332.448267] Lustre: Skipped 2 previous similar messages [ 7334.089599] Lustre: server umount lustre-MDT0000 complete [ 7334.885758] LustreError: 170885:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7334.904722] LustreError: 170885:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7337.537243] LustreError: 172519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7337.835279] LustreError: 170136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746642 with bad export cookie 4099509864737355624 [ 7337.842744] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7337.855515] LustreError: 170136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7340.003370] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7340.005857] LustreError: 170151:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.021099] Lustre: Skipped 1 previous similar message [ 7340.025977] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7342.679991] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.31@tcp (stopping) [ 7347.773342] LustreError: 170150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7347.773864] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.31@tcp (stopping) [ 7347.797304] LustreError: 170150:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7347.817212] Lustre: Skipped 1 previous similar message [ 7352.287181] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7352.646689] Lustre: server umount lustre-MDT0001 complete [ 7359.313606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7371.794445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7372.038583] LustreError: 174020:0:(llog.c:1655:llog_backup()) MGC192.168.203.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7372.046079] Lustre: 174020:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7373.370766] LustreError: 174025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7373.403996] LustreError: 174025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7393.249518] LustreError: 174025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7393.267106] LustreError: 174025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 7428.067127] LustreError: 171986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7428.098369] LustreError: 171986:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [ 7433.183740] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7433.196108] Lustre: 174020:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.131@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7474.173175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7479.283867] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7479.305857] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7488.271420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.576240] LustreError: 174027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.589963] LustreError: 174027:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 45 previous similar messages [ 7502.828428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7509.314288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7509.392696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 7514.254506] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7514.609322] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7522.031707] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7523.609674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7530.117507] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7531.986760] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7554.551995] Lustre: server umount lustre-MDT0000 complete [ 7555.553668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7555.557256] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7555.587519] Lustre: Skipped 1 previous similar message [ 7558.672468] LustreError: 170136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746863 with bad export cookie 4099509864737362330 [ 7558.684900] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7558.694234] LustreError: 170136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7559.182228] Lustre: server umount lustre-MDT0001 complete [ 7574.334845] Lustre: server umount lustre-OST0000 complete [ 7597.573598] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 7608.468574] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 7642.572788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7642.807458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7642.831837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7642.913933] Lustre: lustre-MDT0000: new disk, initializing [ 7642.998978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7643.010399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7646.813701] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7656.814192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7656.954669] Lustre: 179127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7657.031653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7657.038351] Lustre: Skipped 1 previous similar message [ 7657.135700] Lustre: lustre-MDT0001: new disk, initializing [ 7657.300368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7657.321705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7661.804991] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7670.779970] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7677.139128] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7684.810994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7685.183277] Lustre: lustre-OST0000: new disk, initializing [ 7685.193848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7685.217540] Lustre: 181095:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7685.359805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7685.370551] Lustre: Skipped 1 previous similar message [ 7692.356078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7692.369628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7692.513444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7693.632633] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7704.879545] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7710.221836] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7710.410960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7714.555497] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7714.796743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7718.372742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7718.376684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7718.389204] Lustre: Skipped 1 previous similar message [ 7718.390279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7718.390284] Lustre: Skipped 1 previous similar message [ 7722.780732] Lustre: server umount lustre-OST0000 complete [ 7728.609409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7728.623216] Lustre: Skipped 1 previous similar message [ 7738.848755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7738.854960] Lustre: Skipped 6 previous similar messages [ 7741.409580] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7741.608411] Lustre: server umount lustre-MDT0000 complete [ 7743.972762] LustreError: 180436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7743.989293] LustreError: 180436:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [ 7745.176070] LustreError: 179117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747049 with bad export cookie 4099509864737379361 [ 7745.179791] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7745.198317] LustreError: 179117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7745.657855] Lustre: server umount lustre-MDT0001 complete [ 7755.319954] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:44:17 (1790747057) [ 7762.555882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7763.059833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7768.173900] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7776.695230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7781.807031] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7789.720577] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7795.880482] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7803.210827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7810.523344] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7818.534725] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7832.588652] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:45:34 (1790747134) [ 7837.674573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7837.674725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7837.676893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7837.691199] Lustre: Skipped 1 previous similar message [ 7837.712036] LustreError: Skipped 1 previous similar message [ 7842.995761] Lustre: server umount lustre-OST0000 complete [ 7861.215928] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7861.683421] Lustre: server umount lustre-MDT0000 complete [ 7866.195737] LustreError: 183489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747170 with bad export cookie 4099509864737380243 [ 7866.202829] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7866.706815] Lustre: server umount lustre-MDT0001 complete [ 7878.718755] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 7881.938861] Key type lgssc unregistered [ 7882.527734] LNet: 187236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7882.542235] LNetError: 187236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7882.578395] LNet: Removed LNI 192.168.203.131@tcp [ 7883.690792] Key type .llcrypt unregistered [ 7883.695140] Key type ._llcrypt unregistered [ 7905.923442] Key type ._llcrypt registered [ 7905.924815] Key type .llcrypt registered [ 7906.065479] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 7906.931265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7906.972315] alg: No test for adler32 (adler32-zlib) [ 7908.498633] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 7908.947217] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 7910.839212] Key type lgssc registered [ 7913.118679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7923.942183] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7923.977863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7939.615611] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7944.735442] LustreError: 188498:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.131@tcp: failed processing log, type 4: rc = -110 [ 7949.855946] LustreError: 188498:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7949.870145] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7949.888220] LustreError: MGC192.168.203.131@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7949.902510] LustreError: 188498:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7949.907501] LustreError: 188498:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7949.911758] LustreError: 188498:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7949.975518] Lustre: server umount lustre-OST0000 complete [ 7949.981575] LustreError: 188498:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7959.610037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7961.121813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7966.231165] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7970.298990] LustreError: 188998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7970.323436] LustreError: 188998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7975.394373] LustreError: 188999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7976.183559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7976.886324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7982.877568] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7990.810160] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7996.054571] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8002.085350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.462749] LustreError: 190816:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8002.483534] Lustre: 190816:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.131@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8002.710750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8009.427491] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8017.187556] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8021.479918] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8021.495631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8021.499123] Lustre: Skipped 2 previous similar messages [ 8021.507771] Lustre: Skipped 1 previous similar message [ 8022.561942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8022.569476] Lustre: Skipped 3 previous similar messages [ 8026.120026] Lustre: server umount lustre-MDT0000 complete [ 8027.622712] LustreError: 188995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8027.647549] LustreError: 188995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8029.832727] LustreError: 188977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747334 with bad export cookie 14767586773640652072 [ 8029.833794] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8029.854586] LustreError: 188977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8030.816054] LustreError: 189728:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000faeb69ad x1877734549648512/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8031.017114] Lustre: server umount lustre-MDT0001 complete [ 8044.572953] Lustre: server umount lustre-OST0000 complete [ 8055.677606] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 8068.857478] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 8107.248551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8107.525389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8107.562721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8107.654604] Lustre: lustre-MDT0000: new disk, initializing [ 8107.724143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8107.750956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8112.104983] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8121.947752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.076326] Lustre: 194342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8122.141078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8122.147329] Lustre: Skipped 1 previous similar message [ 8122.225211] Lustre: lustre-MDT0001: new disk, initializing [ 8122.318493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8122.359813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8122.369941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8126.389785] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8134.893780] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8139.902764] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8146.132911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8146.324668] Lustre: lustre-OST0000: new disk, initializing [ 8146.331556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8146.338504] Lustre: 196308:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8146.425506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8151.980960] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8153.154658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8153.164599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8153.252891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8160.084685] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8163.972645] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8164.124296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8168.546410] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8168.752804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8174.049033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8174.050808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.059221] LustreError: Skipped 1 previous similar message [ 8174.059879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8174.082586] Lustre: Skipped 1 previous similar message [ 8174.099459] Lustre: Skipped 1 previous similar message [ 8176.832417] Lustre: server umount lustre-OST0000 complete [ 8183.776395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8183.782838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.797340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8186.260647] Lustre: server umount lustre-MDT0000 complete [ 8189.412090] LustreError: 194351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8189.440528] LustreError: 194351:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8189.950855] LustreError: 195136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747494 with bad export cookie 14767586773640652793 [ 8189.954484] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8189.960201] LustreError: 195136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8190.232291] Lustre: server umount lustre-MDT0001 complete [ 8199.041611] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:51:41 (1790747501) [ 8216.554154] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 8219.202496] Key type lgssc unregistered [ 8219.502832] LNet: 199251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8219.513024] LNetError: 199251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8219.530160] LNet: Removed LNI 192.168.203.131@tcp [ 8220.354197] Key type .llcrypt unregistered [ 8220.356184] Key type ._llcrypt unregistered [ 8236.869342] Key type ._llcrypt registered [ 8236.871788] Key type .llcrypt registered [ 8236.991898] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 8237.865791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8237.936591] alg: No test for adler32 (adler32-zlib) [ 8239.007649] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 8239.281872] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 8240.968343] Key type lgssc registered [ 8242.088549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8249.578637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8249.619931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8251.176900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8255.680931] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8260.587981] LustreError: 200483:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8260.609573] LustreError: 200483:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8264.152746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8264.604330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8268.880181] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8275.680047] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8282.940640] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8292.331912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8292.720081] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8292.732272] LustreError: 202349:0:(llog.c:1696:llog_backup()) MGC192.168.203.131@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8292.746287] Lustre: 202349:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.131@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8292.957821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8300.684459] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8308.715880] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8314.852132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8314.860928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8314.871880] Lustre: Skipped 2 previous similar messages [ 8314.882825] Lustre: Skipped 2 previous similar messages [ 8317.763561] Lustre: server umount lustre-MDT0000 complete [ 8319.971339] LustreError: 200483:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8319.983479] LustreError: 200483:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8321.436606] LustreError: 200463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747626 with bad export cookie 4155297223599976886 [ 8321.443032] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8321.677941] Lustre: server umount lustre-MDT0001 complete [ 8336.390440] Lustre: server umount lustre-OST0000 complete [ 8347.039112] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 8355.494385] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 8390.664844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8390.803353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8390.831624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8390.896458] Lustre: lustre-MDT0000: new disk, initializing [ 8390.956912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8390.975436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8394.812918] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8404.547303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.630190] Lustre: 205878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8404.658105] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8404.660607] Lustre: Skipped 1 previous similar message [ 8404.663157] LustreError: 206659:0:(llog.c:1696:llog_backup()) MGC192.168.203.131@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8404.668265] LustreError: 206659:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8404.673324] Lustre: 206659:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.131@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8404.688210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8404.691889] Lustre: Skipped 1 previous similar message [ 8404.745133] Lustre: lustre-MDT0001: new disk, initializing [ 8404.827666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8404.858988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8404.866606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8408.965252] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8419.225977] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8425.456248] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8432.278996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8432.572990] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8432.581188] Lustre: Skipped 1 previous similar message [ 8432.584849] LustreError: 207820:0:(llog.c:1696:llog_backup()) MGC192.168.203.131@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8432.595743] LustreError: 207820:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8432.607670] Lustre: 207820:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.131@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8432.614302] Lustre: 207820:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8432.657730] Lustre: lustre-OST0000: new disk, initializing [ 8432.663556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8432.674594] Lustre: 207845:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8432.773269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8438.857170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8438.875938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8439.011755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8439.268356] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8449.555802] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8454.812685] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8455.126049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8460.195290] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8460.576445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8464.872138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8464.874407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.886306] LustreError: Skipped 1 previous similar message [ 8464.887633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8464.932964] Lustre: Skipped 1 previous similar message [ 8469.985813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8469.995133] Lustre: Skipped 2 previous similar messages [ 8475.113995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8475.133583] Lustre: Skipped 1 previous similar message [ 8477.151137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8477.250017] Lustre: server umount lustre-OST0000 complete [ 8481.760275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8481.770634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8481.786385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8485.347182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8488.025451] Lustre: server umount lustre-MDT0000 complete [ 8490.465526] LustreError: 205890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8490.486772] LustreError: 205890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8492.005573] LustreError: 205871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747796 with bad export cookie 4155297223599977600 [ 8492.007244] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8492.018158] LustreError: 205871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8492.600851] Lustre: server umount lustre-MDT0001 complete [ 8502.189593] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:56:44 (1790747804) [ 8520.280535] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 8530.151886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8530.509604] LustreError: 210973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8530.551659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8534.748547] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8536.033562] LustreError: 210974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8542.585269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.614838] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8550.318343] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8557.690472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8563.839764] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8572.447054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8572.616661] Lustre: lustre-OST0001: new disk, initializing [ 8572.627394] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8572.639658] Lustre: 213279:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8572.727937] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8572.733960] Lustre: Skipped 2 previous similar messages [ 8578.938797] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8581.233439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8581.244290] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8581.332721] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8590.903494] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8599.456858] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8602.003942] Lustre: DEBUG MARKER: ADD OST3 [ 8606.806387] loop: module loaded [ 8606.905068] loop0: detected capacity change from 0 to 204800000 [ 8607.190787] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8607.215500] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8607.255116] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8613.934982] loop0: detected capacity change from 0 to 204800000 [ 8614.321965] Lustre: DEBUG MARKER: STOP OST3 [ 8620.059676] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8620.358176] Lustre: lustre-OST0003: new disk, initializing [ 8620.365167] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8620.370838] Lustre: 215317:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8621.921795] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8622.054193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8622.073154] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8622.153340] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8623.890502] Lustre: Failing over lustre-OST0003 [ 8623.967392] LustreError: 215457:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8623.999821] Lustre: server umount lustre-OST0003 complete [ 8627.239189] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8627.256430] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8627.305922] Lustre: Skipped 1 previous similar message [ 8627.315886] LustreError: 212457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8627.315900] LustreError: 212457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8630.418401] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8645.930993] Lustre: DEBUG MARKER: START OST3 again [ 8647.688351] LustreError: 212456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8647.750316] LustreError: 212456:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 8656.163840] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8657.116762] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8657.128845] Lustre: Skipped 1 previous similar message [ 8657.169210] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8657.977344] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8658.330414] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8658.336759] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8666.912738] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8691.941442] Lustre: Failing over lustre-OST0003 [ 8692.200945] Lustre: server umount lustre-OST0003 complete [ 8693.228685] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8693.230289] LustreError: 212457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8693.238292] Lustre: Skipped 1 previous similar message [ 8693.260076] LustreError: 212457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8713.704504] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8713.723813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8713.726487] Lustre: Skipped 3 previous similar messages [ 8713.767502] Lustre: Skipped 5 previous similar messages [ 8718.600969] Lustre: server umount lustre-MDT0000 complete [ 8722.742521] LustreError: 210954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748027 with bad export cookie 4155297223599978489 [ 8722.748722] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8722.765286] LustreError: 210954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8733.104456] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8733.116670] Lustre: Skipped 5 previous similar messages [ 8737.247574] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8737.690700] Lustre: server umount lustre-MDT0001 complete [ 8742.872481] Lustre: server umount lustre-OST0000 complete [ 8747.429502] Lustre: server umount lustre-OST0001 complete [ 8754.693629] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 8765.265669] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 8809.860462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8810.097241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8810.127430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8810.216829] Lustre: lustre-MDT0000: new disk, initializing [ 8810.308106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8810.331335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8815.620729] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8828.819286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8829.059614] Lustre: 220910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8829.139343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8829.146459] Lustre: Skipped 1 previous similar message [ 8829.229458] Lustre: lustre-MDT0001: new disk, initializing [ 8829.378663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8829.399811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8834.772913] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8845.744560] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8852.050867] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8858.975756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8859.324550] Lustre: 222879:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8860.810921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8860.934823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8865.819748] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8875.045567] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8879.678280] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8879.849290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8884.913608] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8885.176789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8891.873369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8891.878979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8891.890035] LustreError: Skipped 1 previous similar message [ 8891.891382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8891.937533] Lustre: Skipped 3 previous similar messages [ 8893.640875] Lustre: server umount lustre-OST0000 complete [ 8901.088699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8904.117967] Lustre: server umount lustre-MDT0000 complete [ 8907.232698] LustreError: 220922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8907.253422] LustreError: 220922:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 8908.118272] LustreError: 220902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748212 with bad export cookie 4155297223599980785 [ 8908.119343] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8908.134508] LustreError: 220902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8908.617349] Lustre: server umount lustre-MDT0001 complete [ 8912.811753] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:03:35 (1790748215) [ 8931.239580] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 8943.038689] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 8982.553742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8982.733599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8982.774938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8982.779892] Lustre: Skipped 1 previous similar message [ 8982.852609] Lustre: lustre-MDT0000: new disk, initializing [ 8982.855707] Lustre: Skipped 1 previous similar message [ 8982.936489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8982.945213] Lustre: Skipped 1 previous similar message [ 8987.885866] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9001.945911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9002.105441] Lustre: 227493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9002.613825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9009.002849] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9021.986585] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9027.753855] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9034.832561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9035.129572] Lustre: 229462:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9036.410191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9036.655621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9042.416988] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9054.822655] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9061.326838] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9061.731879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9069.030373] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9069.442443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9072.621506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9072.625731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9072.633190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9072.633203] Lustre: Skipped 4 previous similar messages [ 9072.690973] Lustre: Skipped 3 previous similar messages [ 9078.542788] Lustre: server umount lustre-OST0000 complete [ 9090.187800] Lustre: server umount lustre-MDT0000 complete [ 9093.088536] LustreError: 227503:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9093.110866] LustreError: 227503:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9095.347276] LustreError: 227485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748400 with bad export cookie 4155297223599981667 [ 9095.349699] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9095.358656] LustreError: 227485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9095.812083] Lustre: server umount lustre-MDT0001 complete [ 9105.223633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9105.764814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9105.772086] Lustre: Skipped 5 previous similar messages [ 9111.184534] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9120.619227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9125.875692] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9136.091659] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9143.683749] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9151.005787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9158.437563] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9166.283933] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9191.400749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9191.408342] LustreError: Skipped 3 previous similar messages [ 9193.366278] Lustre: server umount lustre-OST0000 complete [ 9201.634863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9201.636545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9201.647730] Lustre: Skipped 9 previous similar messages [ 9201.658426] Lustre: Skipped 5 previous similar messages [ 9211.359405] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9211.674756] Lustre: server umount lustre-MDT0000 complete [ 9215.316359] LustreError: 231661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748520 with bad export cookie 4155297223599982549 [ 9215.329583] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9215.339059] LustreError: 231661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9215.826432] Lustre: server umount lustre-MDT0001 complete [ 9304.538814] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 9307.659880] Key type lgssc unregistered [ 9307.994815] LNet: 235366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9308.004543] LNetError: 235366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9308.028058] LNet: Removed LNI 192.168.203.131@tcp [ 9309.053242] Key type .llcrypt unregistered [ 9309.055473] Key type ._llcrypt unregistered [ 9338.863143] Key type ._llcrypt registered [ 9338.866761] Key type .llcrypt registered [ 9339.017235] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 9355.854867] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 9357.006371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9357.056866] alg: No test for adler32 (adler32-zlib) [ 9358.141807] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 9358.417871] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 9360.103182] Key type lgssc registered [ 9361.473415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9401.038916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9401.074425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9402.385987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9402.459535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9402.582745] Lustre: lustre-MDT0000: new disk, initializing [ 9402.810989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9402.837267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9408.384187] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9420.987933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9421.200843] Lustre: 238740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9421.287952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9421.295399] Lustre: Skipped 1 previous similar message [ 9421.386187] Lustre: lustre-MDT0001: new disk, initializing [ 9421.508393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9421.548658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9421.563076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9426.651548] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9437.310295] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9444.868486] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9452.445979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9452.679638] Lustre: lustre-OST0000: new disk, initializing [ 9452.689224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9452.704124] Lustre: 240707:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9452.838211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9459.312841] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9461.306038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9461.324807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9461.419701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9469.220256] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9473.554351] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9473.806152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9478.085417] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9478.314314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9482.211860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9482.216509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9482.223089] LustreError: Skipped 1 previous similar message [ 9482.226753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9482.248726] Lustre: Skipped 1 previous similar message [ 9486.635885] Lustre: server umount lustre-OST0000 complete [ 9492.455291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9492.459981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9492.464092] Lustre: Skipped 1 previous similar message [ 9492.484187] Lustre: Skipped 2 previous similar messages [ 9497.072757] Lustre: server umount lustre-MDT0000 complete [ 9500.992410] LustreError: 238729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748805 with bad export cookie 1127974465599546223 [ 9501.002867] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9501.008973] LustreError: 238729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9501.572185] Lustre: server umount lustre-MDT0001 complete [ 9522.900941] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 9534.558605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9534.946856] LustreError: 243641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9534.961481] LustreError: 243641:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9535.001865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9539.572625] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9540.076733] LustreError: 243642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9545.186418] LustreError: 243641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9548.285485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9548.790143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9553.775759] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9556.586449] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9564.218659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9564.652426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9571.397308] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9580.087860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9580.247869] Lustre: lustre-OST0001: new disk, initializing [ 9580.251450] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9580.257345] Lustre: 245949:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9587.317384] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9587.363166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9587.375502] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9587.512749] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9599.385975] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9610.310833] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9624.310585] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:15:27 (1790748927) [ 9628.645125] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9628.654309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9628.658048] Lustre: Skipped 2 previous similar messages [ 9628.671708] Lustre: Skipped 3 previous similar messages [ 9632.892767] Lustre: server umount lustre-MDT0000 complete [ 9633.771211] LustreError: 243638:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9633.779837] LustreError: 243638:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9636.663441] LustreError: 243621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748941 with bad export cookie 1127974465599547105 [ 9636.678194] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9636.679170] LustreError: 243621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9637.044262] Lustre: server umount lustre-MDT0001 complete [ 9650.974898] Lustre: server umount lustre-OST0000 complete [ 9665.146680] Lustre: server umount lustre-OST0001 complete [ 9671.453628] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 9682.425229] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 9723.246169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9723.494118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9723.546877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9723.677568] Lustre: lustre-MDT0000: new disk, initializing [ 9723.850628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9723.860834] Lustre: Skipped 1 previous similar message [ 9723.899585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9729.340618] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9741.059396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9741.161798] Lustre: 249897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9741.224475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9741.228261] Lustre: Skipped 1 previous similar message [ 9741.323298] Lustre: lustre-MDT0001: new disk, initializing [ 9741.520971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9741.530966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9746.591381] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9757.207047] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9763.209309] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9771.100036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9771.514816] Lustre: lustre-OST0000: new disk, initializing [ 9771.525810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9771.548275] Lustre: 251868:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9771.672423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9771.676793] Lustre: Skipped 1 previous similar message [ 9772.890769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9772.903022] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9773.030900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9778.110248] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9787.619771] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9793.164329] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9793.384876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9798.527881] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9798.862740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9803.753625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9803.759076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9803.765354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9803.803838] Lustre: Skipped 2 previous similar messages [ 9807.444776] Lustre: server umount lustre-OST0000 complete [ 9812.960211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9812.968624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9812.979849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9812.987107] Lustre: Skipped 1 previous similar message [ 9817.934830] Lustre: server umount lustre-MDT0000 complete [ 9819.114613] LustreError: 250784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9819.136056] LustreError: 250784:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9822.409539] LustreError: 250695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749127 with bad export cookie 1127974465599549233 [ 9822.413778] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9822.420657] LustreError: 250695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9822.758137] Lustre: server umount lustre-MDT0001 complete [ 9831.852354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9832.360921] LustreError: 254084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9837.120779] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9847.209885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9847.634459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9847.644118] Lustre: Skipped 1 previous similar message [ 9852.784347] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9861.045420] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9866.734354] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9874.629630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9882.086459] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9889.019858] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10036.195863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.240911] Lustre: Skipped 1 previous similar message [10036.257202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10036.260141] Lustre: Skipped 2 previous similar messages [10039.499948] Lustre: server umount lustre-OST0000 complete [10049.943577] Lustre: server umount lustre-MDT0000 complete [10051.560284] LustreError: 254083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10051.588399] LustreError: 254083:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10053.709434] LustreError: 254066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749358 with bad export cookie 1127974465599550122 [10053.711508] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10053.719420] LustreError: 254066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10054.142753] Lustre: server umount lustre-MDT0001 complete [10157.054135] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [10159.796923] Key type lgssc unregistered [10160.095872] LNet: 257690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10160.112767] LNetError: 257690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10160.143258] LNet: Removed LNI 192.168.203.131@tcp [10161.173786] Key type .llcrypt unregistered [10161.178305] Key type ._llcrypt unregistered [10186.979673] Key type ._llcrypt registered [10186.982773] Key type .llcrypt registered [10187.073562] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [10202.287807] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [10203.328394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10203.337967] alg: No test for adler32 (adler32-zlib) [10204.366471] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [10204.672303] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [10206.393724] Key type lgssc registered [10207.748112] Lustre: Echo OBD driver; http://www.lustre.org/ [10258.868985] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [10269.555850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10269.598958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10270.797451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10270.840428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10270.919366] Lustre: lustre-MDT0000: new disk, initializing [10271.016402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10271.038202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10275.398190] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10286.581058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10286.735782] Lustre: 261780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10286.848317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10286.855500] Lustre: Skipped 1 previous similar message [10286.987387] Lustre: lustre-MDT0001: new disk, initializing [10287.146842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10287.202928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10287.218267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10291.690491] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10297.069314] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10303.978282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10304.248634] Lustre: lustre-OST0000: new disk, initializing [10304.252316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10304.265311] Lustre: 263413:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10304.383047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10308.104379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10308.118304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10308.219307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10311.332988] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10323.043163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10323.272094] Lustre: lustre-OST0001: new disk, initializing [10323.275288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10323.281782] Lustre: 264283:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10323.419698] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10329.863759] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10331.750675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10331.765446] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10331.862760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10342.208288] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10349.990375] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10363.872207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10363.881855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10363.893138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10367.970576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10367.988599] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10368.015190] Lustre: Skipped 1 previous similar message [10369.323770] Lustre: server umount lustre-MDT0000 complete [10373.365215] LustreError: 265052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749678 with bad export cookie 12658002942557242133 [10373.372099] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10373.377963] LustreError: 265052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10373.772850] Lustre: server umount lustre-MDT0001 complete [10387.598531] Lustre: server umount lustre-OST0000 complete [10402.077230] Lustre: server umount lustre-OST0001 complete [10417.914431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10428.759172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10436.780221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10446.655247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10468.483127] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [10479.206663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10479.233378] Lustre: lustre-MDT0000: reset Object Index mappings [10479.610318] LustreError: 269296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10479.629504] LustreError: 269296:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10479.689395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10484.411452] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10484.705522] LustreError: 269297:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10489.848158] LustreError: 269296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10493.637322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10493.668845] Lustre: lustre-MDT0001: reset Object Index mappings [10494.119305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10499.158914] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10502.643979] Lustre: 270439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10510.850350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10511.420277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10518.227421] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10518.575843] LustreError: 270792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10523.619203] LustreError: 270793:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10524.678252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10528.328621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10533.899894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10536.916948] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10544.889867] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10550.012595] Lustre: 272312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10559.457112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10559.467539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10559.479762] Lustre: Skipped 3 previous similar messages [10559.483344] Lustre: Skipped 4 previous similar messages [10562.716975] Lustre: server umount lustre-MDT0000 complete [10564.588468] LustreError: 269292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10564.623463] LustreError: 269292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10566.705747] LustreError: 269277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749871 with bad export cookie 12658002942557248923 [10566.710234] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10566.715921] LustreError: 269277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10567.117627] Lustre: server umount lustre-MDT0001 complete [10582.227887] Lustre: server umount lustre-OST0000 complete [10586.086467] Lustre: 259278:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790749874/real 1790749874] req@ffff94504a908700 x1877736957282944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790749890 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10586.150745] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10586.167982] Lustre: Skipped 1 previous similar message [10587.114797] Lustre: server umount lustre-OST0001 complete [10611.953503] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:31:54 (1790749914) [10631.348150] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [10646.166538] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [10688.621086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10688.876713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10688.896532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10688.956563] Lustre: lustre-MDT0000: new disk, initializing [10689.016306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10689.022220] Lustre: Skipped 1 previous similar message [10689.037047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10694.423774] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10707.165217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10707.363548] Lustre: 276586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10707.518460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10707.529894] Lustre: Skipped 1 previous similar message [10707.778347] Lustre: lustre-MDT0001: new disk, initializing [10708.001642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10708.024601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10713.244503] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10724.564924] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10731.079925] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10739.039336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10739.280196] Lustre: lustre-OST0000: new disk, initializing [10739.286345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10739.297783] Lustre: 278556:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10740.998622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10741.027650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10741.127710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10746.565703] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10756.606293] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10761.398644] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10761.654961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10767.474667] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10767.678797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10771.938293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10771.942214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10771.950174] LustreError: Skipped 1 previous similar message [10771.951653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10771.951662] Lustre: Skipped 2 previous similar messages [10771.978875] Lustre: Skipped 1 previous similar message [10775.892160] Lustre: server umount lustre-OST0000 complete [10779.617450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10779.628857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10779.634288] Lustre: Skipped 1 previous similar message [10782.180108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10782.202701] Lustre: Skipped 1 previous similar message [10785.431766] Lustre: server umount lustre-MDT0000 complete [10787.296269] LustreError: 276598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.313945] LustreError: 276598:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10788.817090] LustreError: 276581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750093 with bad export cookie 12658002942557251016 [10788.822283] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10788.825706] LustreError: 276581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10789.138834] Lustre: server umount lustre-MDT0001 complete [10806.988496] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [10817.176195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10817.688077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10817.693854] Lustre: Skipped 2 previous similar messages [10822.231902] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10823.137065] LustreError: 281492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10823.157254] LustreError: 281492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10830.616041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10835.438156] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10838.196981] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10845.195752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10851.268952] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10860.016280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10860.178726] Lustre: lustre-OST0001: new disk, initializing [10860.182442] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10860.190367] Lustre: 283798:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10862.011131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10862.018133] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10862.169830] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10866.554684] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10876.510276] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10880.522818] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10886.077155] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10886.098129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10896.493798] Lustre: server umount lustre-MDT0000 complete [10897.890154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10897.901761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10897.918995] LustreError: 281493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10897.944776] LustreError: 281493:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10900.030798] LustreError: 281473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750204 with bad export cookie 12658002942557251898 [10900.036272] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10900.045540] LustreError: 281473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10900.439329] Lustre: server umount lustre-MDT0001 complete [10914.455175] Lustre: server umount lustre-OST0000 complete [10928.423464] Lustre: server umount lustre-OST0001 complete [10947.703878] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [10958.347460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10958.940746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10958.943847] Lustre: Skipped 3 previous similar messages [10963.837847] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10973.964180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10979.083278] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10982.165213] Lustre: 288285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10989.656310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10997.006689] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11003.368749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11007.236798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11012.603233] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11013.913863] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11022.782289] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11027.041892] Lustre: 290163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11032.314962] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11032.558705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11046.558701] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:39:09 (1790750349) [11063.780874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11063.799689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11063.809674] Lustre: Skipped 5 previous similar messages [11063.825768] Lustre: Skipped 5 previous similar messages [11068.302615] Lustre: server umount lustre-MDT0000 complete [11068.903526] LustreError: 287146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11068.922967] LustreError: 287146:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [11072.223843] LustreError: 287125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750376 with bad export cookie 12658002942557258303 [11072.234658] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11072.238663] LustreError: 287125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11078.763358] Lustre: server umount lustre-MDT0001 complete [11089.381700] Lustre: server umount lustre-OST0000 complete [11093.675992] Lustre: server umount lustre-OST0001 complete [11100.651203] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [11113.174435] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [11163.512696] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [11174.109978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11174.386928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11174.416256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11174.475146] Lustre: lustre-MDT0000: new disk, initializing [11174.558097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11179.291339] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11190.331533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.459291] Lustre: 294693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11190.777294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11195.807730] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11209.092250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11209.456728] Lustre: 296327:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11211.582219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11216.851444] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11227.457254] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11227.627067] Lustre: 297196:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11227.713382] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11227.721235] Lustre: Skipped 6 previous similar messages [11229.726786] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11229.732899] Lustre: Skipped 1 previous similar message [11229.879142] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11235.614477] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11247.513873] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11252.375258] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11252.382476] Lustre: Skipped 1 previous similar message [11268.616418] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:42:51 (1790750571) [11271.149018] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11271.173534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11271.181024] Lustre: Skipped 4 previous similar messages [11271.209407] Lustre: Skipped 7 previous similar messages [11272.962507] Lustre: server umount lustre-MDT0000 complete [11276.819498] LustreError: 294686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750581 with bad export cookie 12658002942557266220 [11276.828033] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11276.843259] LustreError: 294686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11277.311922] Lustre: server umount lustre-MDT0001 complete [11292.021231] Lustre: server umount lustre-OST0000 complete [11306.007565] Lustre: server umount lustre-OST0001 complete [11315.516540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11320.982984] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11325.420430] LustreError: 299446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11325.446742] LustreError: 299446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11330.850300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11336.528092] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11344.620242] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11351.814103] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11359.795503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11366.848542] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11375.736510] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11396.678621] loop1: detected capacity change from 0 to 204800000 [11396.977673] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11396.994455] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11397.005987] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11402.830720] loop1: detected capacity change from 0 to 204800000 [11408.904115] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11409.015341] LustreError: 299435:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs15448: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11409.027516] LustreError: 299435:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs15448 [11409.031458] LustreError: 299435:0:(mgs_handler.c:517:mgs_target_reg()) fs15448-MDT0001: Error checking target index: rc = -13 [11409.040327] LustreError: fs15448-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11409.050519] LustreError: 303166:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11409.064263] LustreError: 303166:0:(tgt_mount.c:2128:server_put_super()) no obd fs15448-MDT0001 [11409.069928] LustreError: 303166:0:(tgt_mount.c:116:server_deregister_mount()) fs15448-MDT0001 not registered [11409.120520] Lustre: server umount fs15448-MDT0001 complete [11409.127598] LustreError: 303166:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11417.942730] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11418.073752] Lustre: Setting parameter fs15448-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15448-MDT0001 [11418.139473] Lustre: srv-fs15448-MDT0001: No data found on store. Initialize space. [11418.144879] Lustre: Skipped 4 previous similar messages [11418.238941] Lustre: fs15448-MDT0001: new disk, initializing [11418.242714] Lustre: Skipped 3 previous similar messages [11418.373074] Lustre: srv-fs15448-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11418.388127] Lustre: 303632:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15448-MDT0001-mdtlov: can't create new llog: rc = -115 [11418.407566] LustreError: 303632:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15448-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11423.246582] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11430.363495] Lustre: server umount fs15448-MDT0001 complete [11439.100064] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11444.414536] loop1: detected capacity change from 0 to 204800000 [11450.814589] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11451.567761] Lustre: 305201:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15448-MDT0001-mdtlov: can't create new llog: rc = -115 [11451.588602] LustreError: 305201:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15448-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11457.476863] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11459.570723] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11463.362323] Lustre: server umount fs15448-MDT0001 complete [11474.928334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11474.936515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11474.942083] Lustre: Skipped 3 previous similar messages [11489.247133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11489.480193] Lustre: server umount lustre-MDT0000 complete [11498.834267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11498.927499] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11504.636375] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11505.207895] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11514.114603] loop1: detected capacity change from 0 to 204800000 [11519.925129] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11520.428337] Lustre: 307650:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15448-MDT0001-mdtlov: can't create new llog: rc = -115 [11520.446390] LustreError: 307650:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15448-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11525.533201] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11530.444927] Lustre: server umount fs15448-MDT0001 complete [11555.809610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11555.819940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11555.826268] Lustre: Skipped 11 previous similar messages [11561.812769] Lustre: server umount lustre-OST0000 complete [11573.543374] Lustre: server umount lustre-MDT0000 complete [11577.429805] LustreError: 299427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750882 with bad export cookie 12658002942557268908 [11577.443243] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11577.448937] LustreError: 299427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11577.878891] Lustre: server umount lustre-MDT0001 complete [11588.918296] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [11591.473868] Key type lgssc unregistered [11591.795499] LNet: 309994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11591.807438] LNetError: 309994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11591.854861] LNet: Removed LNI 192.168.203.131@tcp [11592.806244] Key type .llcrypt unregistered [11592.811873] Key type ._llcrypt unregistered [11614.280980] Key type ._llcrypt registered [11614.286561] Key type .llcrypt registered [11614.425808] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [11615.625249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11615.676062] alg: No test for adler32 (adler32-zlib) [11616.871841] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [11617.116448] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [11618.775217] Key type lgssc registered [11619.889327] Lustre: Echo OBD driver; http://www.lustre.org/ [11628.575052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11628.601736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11630.216697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11635.504580] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11639.784074] LustreError: 311286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11639.803740] LustreError: 311286:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11644.901984] LustreError: 311287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11645.282577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11645.712226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11650.157833] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11658.478181] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11664.551889] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11671.275137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11671.763417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11674.862185] LustreError: 313118:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.493920] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11680.236679] LustreError: 313118:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11685.351375] LustreError: 313118:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11685.916865] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11694.149323] LustreError: 313118:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.172625] LustreError: 313118:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11708.169239] loop1: detected capacity change from 0 to 204800000 [11708.360170] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11708.368799] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11708.385716] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11710.952906] LustreError: 313115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11710.971202] LustreError: 313115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11714.061643] loop1: detected capacity change from 0 to 204800000 [11719.111716] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11719.345309] Lustre: fs11562-OST0000: new disk, initializing [11719.350231] Lustre: srv-fs11562-OST0000: No data found on store. Initialize space. [11719.357221] Lustre: 315022:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11719.442246] Lustre: fs11562-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11725.566131] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11734.489441] Lustre: server umount fs11562-OST0000 complete [11745.340677] LustreError: 313115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11745.364944] LustreError: 313115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [11747.857521] loop1: detected capacity change from 0 to 204800000 [11753.589577] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11753.893283] Lustre: fs11562-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11757.054228] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11759.795733] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11769.565441] loop2: detected capacity change from 0 to 204800000 [11769.975370] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11769.984773] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11770.001560] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11772.397316] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11775.248491] loop2: detected capacity change from 0 to 204800000 [11780.582520] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11780.682697] LustreError: fs11562-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11780.691621] LustreError: fs11562-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11780.698424] LustreError: 318062:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11780.703518] LustreError: 318062:0:(tgt_mount.c:2128:server_put_super()) no obd fs11562-OST0001 [11780.708812] LustreError: 318062:0:(tgt_mount.c:116:server_deregister_mount()) fs11562-OST0001 not registered [11780.737086] Lustre: server umount fs11562-OST0001 complete [11780.740359] LustreError: 318062:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11790.561083] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11790.810923] Lustre: fs11562-OST0001: new disk, initializing [11790.814198] Lustre: srv-fs11562-OST0001: No data found on store. Initialize space. [11790.828310] Lustre: 318613:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11790.919188] Lustre: fs11562-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11797.183321] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11805.533405] Lustre: server umount fs11562-OST0000 complete [11811.899651] LustreError: 313118:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11811.926270] LustreError: 313118:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [11814.655276] Lustre: server umount fs11562-OST0001 complete [11825.769853] loop1: detected capacity change from 0 to 204800000 [11830.545867] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11830.872491] Lustre: fs11562-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11836.994740] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11843.796777] loop2: detected capacity change from 0 to 204800000 [11850.190360] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11857.227397] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11868.383513] Lustre: server umount fs11562-OST0000 complete [11878.564361] Lustre: server umount fs11562-OST0001 complete [11895.264476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11895.280497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11895.282648] LustreError: Skipped 1 previous similar message [11895.286507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11895.294098] Lustre: Skipped 1 previous similar message [11900.399781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11900.411787] Lustre: Skipped 1 previous similar message [11905.507295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11905.516815] Lustre: Skipped 2 previous similar messages [11907.040858] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11907.231934] Lustre: server umount lustre-OST0000 complete [11912.160503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11912.173545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11912.195478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11915.747650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11917.142563] Lustre: server umount lustre-MDT0000 complete [11921.905794] LustreError: 313140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790751226 with bad export cookie 7484460988102707553 [11921.908681] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11921.917309] LustreError: 313140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11922.492833] Lustre: server umount lustre-MDT0001 complete [11934.254985] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [11936.908693] Key type lgssc unregistered [11937.282657] LNet: 324538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11937.288050] LNetError: 324538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11937.301309] LNet: Removed LNI 192.168.203.131@tcp [11938.338155] Key type .llcrypt unregistered [11938.341065] Key type ._llcrypt unregistered [11971.907753] Key type ._llcrypt registered [11971.915824] Key type .llcrypt registered [11972.076977] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [11989.219709] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [11990.646589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11990.681577] alg: No test for adler32 (adler32-zlib) [11991.725439] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [11991.901744] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [11993.544278] Key type lgssc registered [11995.297558] Lustre: Echo OBD driver; http://www.lustre.org/ [12036.877135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12036.932827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12038.153820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12038.194765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12038.321610] Lustre: lustre-MDT0000: new disk, initializing [12038.432973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12038.456806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12043.427048] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12054.599962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12054.742469] Lustre: 328414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12054.793773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12054.797700] Lustre: Skipped 1 previous similar message [12054.900978] Lustre: lustre-MDT0001: new disk, initializing [12054.997776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12055.044626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12055.054703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12059.632494] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12068.409831] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12074.379833] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12081.081195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12081.309784] Lustre: lustre-OST0000: new disk, initializing [12081.313534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12081.320245] Lustre: 330382:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12081.419979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12088.095112] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12089.490273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12089.511764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12089.601277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12099.320796] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12101.352669] Lustre: Mounted lustre-client - version 2.17.59_3_g94a7b9b [12102.384633] Lustre: Unmounted lustre-client [12115.424792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12115.427412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12115.435616] LustreError: Skipped 1 previous similar message [12115.436616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12115.490058] Lustre: Skipped 1 previous similar message [12116.960784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12116.970841] Lustre: Skipped 2 previous similar messages [12119.151914] Lustre: server umount lustre-OST0000 complete [12126.692683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12126.705606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12126.734453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12129.977603] Lustre: server umount lustre-MDT0000 complete [12134.337089] LustreError: 328405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790751439 with bad export cookie 17239307738052006492 [12134.340930] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12134.350957] LustreError: 328405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12134.630971] Lustre: server umount lustre-MDT0001 complete [12151.353720] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [12155.400530] Key type lgssc unregistered [12160.872191] LNet: 332541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12160.888531] LNetError: 332541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12232.613835] LNet: Removed LNI 192.168.203.131@tcp [12233.592707] Key type .llcrypt unregistered [12233.595282] Key type ._llcrypt unregistered [12255.518853] Key type ._llcrypt registered [12255.520921] Key type .llcrypt registered [12255.618937] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [12272.776380] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [12273.352796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12273.420616] alg: No test for adler32 (adler32-zlib) [12274.630928] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [12274.888346] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [12276.559225] Key type lgssc registered [12277.947091] Lustre: Echo OBD driver; http://www.lustre.org/ [12317.675774] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:00:20 (1790751620) [12325.119248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12325.144771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12326.454726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12326.500447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12326.630561] Lustre: lustre-MDT0000: new disk, initializing [12326.718806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12326.734664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12331.718656] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12343.134037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12343.278080] Lustre: 335781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12343.385711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12343.393379] Lustre: Skipped 1 previous similar message [12343.516299] Lustre: lustre-MDT0001: new disk, initializing [12343.631520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12343.691811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12343.708679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12349.066617] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12360.290324] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12367.756163] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12375.476323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12375.843757] Lustre: lustre-OST0000: new disk, initializing [12375.847546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12375.860161] Lustre: 337760:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12375.964529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12382.824228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12382.830686] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12382.853610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12382.965524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12393.371600] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12413.930270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12413.934262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12413.951360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12413.952887] LustreError: Skipped 1 previous similar message [12413.976132] Lustre: Skipped 1 previous similar message [12415.275985] Lustre: server umount lustre-OST0000 complete [12420.580972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12420.591800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12420.608598] Lustre: Skipped 1 previous similar message [12420.619081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12424.167344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12424.169357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12426.078732] Lustre: server umount lustre-MDT0000 complete [12430.817777] LustreError: 335772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790751735 with bad export cookie 11478225528594575625 [12430.828123] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12430.841930] LustreError: 335772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12431.496675] Lustre: server umount lustre-MDT0001 complete [12444.344802] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [12448.075357] Key type lgssc unregistered [12448.447669] LNet: 339670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12448.455947] LNetError: 339670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12449.511120] LNet: Removed LNI 192.168.203.131@tcp [12450.433811] Key type .llcrypt unregistered [12450.438497] Key type ._llcrypt unregistered [12473.408267] Key type ._llcrypt registered [12473.410595] Key type .llcrypt registered [12473.499321] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [12474.644077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12474.832432] alg: No test for adler32 (adler32-zlib) [12476.004829] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [12476.288819] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [12478.015546] Key type lgssc registered [12479.280029] Lustre: Echo OBD driver; http://www.lustre.org/ [12488.343930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12488.393488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12490.000955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12495.776593] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12499.429585] LustreError: 340960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12499.459888] LustreError: 340960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12504.571409] LustreError: 340961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12506.019404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12506.413540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12510.934518] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12518.736323] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12525.318959] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12532.579224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12532.968294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12540.657659] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12548.625759] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12562.116364] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12563.589590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12568.032134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12568.043398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12568.072995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12571.730317] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.31@tcp (stopping) [12572.136127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12572.148457] Lustre: Skipped 1 previous similar message [12574.084855] Lustre: server umount lustre-MDT0000 complete [12576.838402] LustreError: 340957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12578.213887] LustreError: 341696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790751882 with bad export cookie 15475406651307936328 [12578.224550] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12581.948670] LustreError: 340957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12581.985494] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.31@tcp (stopping) [12581.990306] LustreError: 340957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12582.017671] Lustre: Skipped 3 previous similar messages [12582.373733] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12584.634453] Lustre: server umount lustre-MDT0001 complete [12589.691995] Lustre: server umount lustre-OST0000 complete [12600.044038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12605.789913] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12608.736675] Lustre: 345143:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.131@tcp [12611.065741] Lustre: server umount MGS complete [12620.189881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12620.781117] LustreError: 345717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.803802] LustreError: 345717:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12620.870727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12625.974584] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12631.011414] LustreError: 345718:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12631.038154] LustreError: 345718:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12636.604058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12636.972450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12642.886291] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12652.548964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12652.851989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12658.723868] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12667.528957] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12669.302817] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12675.713569] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12677.313930] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12681.697042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12681.706088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12681.711520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12681.711531] Lustre: Skipped 2 previous similar messages [12681.752085] Lustre: Skipped 1 previous similar message [12683.838885] Lustre: server umount lustre-OST0000 complete [12688.352478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12688.360559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12688.387755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12688.393049] Lustre: Skipped 1 previous similar message [12693.757979] Lustre: server umount lustre-MDT0000 complete [12697.061326] LustreError: 345718:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12697.104992] LustreError: 345718:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12697.750704] LustreError: 345698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790752002 with bad export cookie 15475406651307937168 [12697.755747] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12697.768045] LustreError: 345698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12698.222640] Lustre: server umount lustre-MDT0001 complete [12709.955616] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [12712.925323] Key type lgssc unregistered [12713.199315] LNet: 349153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12713.207856] LNetError: 349153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12713.227941] LNet: Removed LNI 192.168.203.131@tcp [12714.451362] Key type .llcrypt unregistered [12714.455879] Key type ._llcrypt unregistered [12734.446436] Key type ._llcrypt registered [12734.448723] Key type .llcrypt registered [12734.538707] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [12750.769784] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [12751.905162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12751.932549] alg: No test for adler32 (adler32-zlib) [12753.264947] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [12753.542957] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [12755.287419] Key type lgssc registered [12756.697409] Lustre: Echo OBD driver; http://www.lustre.org/ [12795.675640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12795.731945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12797.069712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12797.126351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12797.256328] Lustre: lustre-MDT0000: new disk, initializing [12797.380655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12797.503324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12802.183865] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12814.227343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12814.370988] Lustre: 352333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12814.437854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12814.444634] Lustre: Skipped 1 previous similar message [12814.535001] Lustre: lustre-MDT0001: new disk, initializing [12814.651936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12814.704140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12814.718110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12819.669417] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12830.593668] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12836.776389] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12844.793841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12845.117969] Lustre: lustre-OST0000: new disk, initializing [12845.123913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12845.140053] Lustre: 354303:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12845.291760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12852.896896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12852.909703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12853.077372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12853.234490] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12864.583737] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12870.705770] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12871.074895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12877.135745] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12877.373483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12883.940902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12883.941497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12883.945817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12883.949730] LustreError: Skipped 1 previous similar message [12883.963910] Lustre: Skipped 1 previous similar message [12886.362272] Lustre: server umount lustre-OST0000 complete [12891.621291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12891.648233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12891.701439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12891.722562] Lustre: Skipped 1 previous similar message [12894.180030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12894.189454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12897.303512] Lustre: server umount lustre-MDT0000 complete [12901.639686] LustreError: 352325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790752206 with bad export cookie 18433205697881740651 [12901.642225] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12901.648186] LustreError: 352325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12902.374567] Lustre: server umount lustre-MDT0001 complete [12912.033615] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:10:14 (1790752214) [12930.178520] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [12946.401975] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [12990.446738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12990.692568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12990.740906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12990.865455] Lustre: lustre-MDT0000: new disk, initializing [12990.977469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12990.992634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12995.544401] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13006.294155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13006.444544] Lustre: 359089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13006.496113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13006.500567] Lustre: Skipped 1 previous similar message [13006.570926] Lustre: lustre-MDT0001: new disk, initializing [13006.665837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13006.706646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13006.724143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13011.264923] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13020.813985] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13026.643374] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13033.897203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13034.280597] Lustre: lustre-OST0000: new disk, initializing [13034.285093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13034.297325] Lustre: 361055:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13034.418580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13040.729252] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13043.766985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13043.780667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13043.879869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13050.675418] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13085.152554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13085.155850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13085.164407] LustreError: Skipped 1 previous similar message [13085.178617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13085.178628] Lustre: Skipped 1 previous similar message [13090.274420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13090.283391] Lustre: Skipped 2 previous similar messages [13091.418620] Lustre: server umount lustre-OST0000 complete [13095.399209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13095.414707] Lustre: Skipped 2 previous similar messages [13100.519060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13100.534060] Lustre: Skipped 3 previous similar messages [13101.616529] Lustre: server umount lustre-MDT0000 complete [13105.008433] LustreError: 359080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790752409 with bad export cookie 18433205697881741533 [13105.020611] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13105.339635] Lustre: server umount lustre-MDT0001 complete [13116.551451] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13119.703267] Key type lgssc unregistered [13120.030851] LNet: 363229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13120.040143] LNetError: 363229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13120.064147] LNet: Removed LNI 192.168.203.131@tcp [13120.973334] Key type .llcrypt unregistered [13120.976811] Key type ._llcrypt unregistered [13142.373197] Key type ._llcrypt registered [13142.375716] Key type .llcrypt registered [13142.476658] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [13143.738463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13143.910577] alg: No test for adler32 (adler32-zlib) [13144.964958] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [13145.178088] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [13146.839219] Key type lgssc registered [13147.950510] Lustre: Echo OBD driver; http://www.lustre.org/ [13155.615979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13155.636277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13157.128858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13161.851354] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13166.562597] LustreError: 364522:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13166.582047] LustreError: 364522:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13171.311037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13171.679983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13171.814146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13176.865979] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13184.601203] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13190.331973] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13197.221572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13197.740760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13204.470495] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13212.180511] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13268.455564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13268.456364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13268.467327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13268.501179] Lustre: Skipped 1 previous similar message [13273.570317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13273.590739] Lustre: Skipped 1 previous similar message [13278.692821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13278.697287] Lustre: Skipped 1 previous similar message [13281.247739] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13281.468390] Lustre: server umount lustre-OST0000 complete [13288.930666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13288.950551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13288.956264] Lustre: Skipped 1 previous similar message [13299.168985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13299.177269] Lustre: Skipped 3 previous similar messages [13300.191343] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13300.433508] Lustre: server umount lustre-MDT0000 complete [13304.069529] LustreError: 364503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790752608 with bad export cookie 15970605470780698960 [13304.072631] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13304.079076] LustreError: 364503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13304.423965] Lustre: server umount lustre-MDT0001 complete [13315.868163] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13318.429836] Key type lgssc unregistered [13318.689092] LNet: 370132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13318.695263] LNetError: 370132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13318.708982] LNet: Removed LNI 192.168.203.131@tcp [13319.479841] Key type .llcrypt unregistered [13319.485382] Key type ._llcrypt unregistered [13411.610780] Key type ._llcrypt registered [13411.612258] Key type .llcrypt registered [13411.740884] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13412.086421] Key type .llcrypt unregistered [13412.091768] Key type ._llcrypt unregistered [13491.098607] Key type ._llcrypt registered [13491.100159] Key type .llcrypt registered [13491.218740] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13491.489338] Key type .llcrypt unregistered [13491.492429] Key type ._llcrypt unregistered [13522.526609] Key type ._llcrypt registered [13522.531335] Key type .llcrypt registered [13522.661658] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13522.961408] Key type .llcrypt unregistered [13522.966968] Key type ._llcrypt unregistered [13557.785805] Key type ._llcrypt registered [13557.787614] Key type .llcrypt registered [13557.894460] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13558.216153] Key type .llcrypt unregistered [13558.224612] Key type ._llcrypt unregistered [13634.104543] Key type ._llcrypt registered [13634.108181] Key type .llcrypt registered [13634.205675] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13634.442461] Key type .llcrypt unregistered [13634.445771] Key type ._llcrypt unregistered [13660.064126] Key type ._llcrypt registered [13660.068023] Key type .llcrypt registered [13660.189792] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13660.549375] Key type .llcrypt unregistered [13660.551237] Key type ._llcrypt unregistered [13690.215485] Key type ._llcrypt registered [13690.218611] Key type .llcrypt registered [13690.368960] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13690.689424] Key type .llcrypt unregistered [13690.693980] Key type ._llcrypt unregistered [13716.968933] Key type ._llcrypt registered [13716.972778] Key type .llcrypt registered [13717.081546] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13717.356421] Key type .llcrypt unregistered [13717.359782] Key type ._llcrypt unregistered [13746.572917] Key type ._llcrypt registered [13746.579566] Key type .llcrypt registered [13746.738960] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13747.061298] Key type .llcrypt unregistered [13747.063727] Key type ._llcrypt unregistered [13773.415525] Key type ._llcrypt registered [13773.417575] Key type .llcrypt registered [13773.549747] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13773.767593] Key type .llcrypt unregistered [13773.770498] Key type ._llcrypt unregistered [13831.153128] Key type ._llcrypt registered [13831.157005] Key type .llcrypt registered [13831.236353] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13831.464278] Key type .llcrypt unregistered [13831.467422] Key type ._llcrypt unregistered [13862.840407] Key type ._llcrypt registered [13862.842779] Key type .llcrypt registered [13862.966107] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13863.386292] Key type .llcrypt unregistered [13863.392658] Key type ._llcrypt unregistered [13894.986265] Key type ._llcrypt registered [13894.988174] Key type .llcrypt registered [13895.123995] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13895.408408] Key type .llcrypt unregistered [13895.414100] Key type ._llcrypt unregistered [13947.691052] Key type ._llcrypt registered [13947.693088] Key type .llcrypt registered [13947.781885] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [13962.977108] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [13964.025671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13964.035855] alg: No test for adler32 (adler32-zlib) [13965.132084] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [13965.407925] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [13967.111459] Key type lgssc registered [13968.360980] Lustre: Echo OBD driver; http://www.lustre.org/ [14002.532740] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [14002.559502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14003.774720] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14003.803562] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14003.860968] Lustre: tst250fs-MDT0000: new disk, initializing [14003.938900] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14003.953568] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14007.951697] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14019.024065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14019.149569] Lustre: 390384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14019.252650] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14019.261331] Lustre: Skipped 1 previous similar message [14019.385373] Lustre: tst250fs-MDT0001: new disk, initializing [14019.493977] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14019.538027] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14019.553472] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14023.526755] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14028.392404] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14034.119895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14034.397748] Lustre: tst250fs-OST0000: new disk, initializing [14034.403962] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14034.410217] Lustre: 392015:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14034.516091] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14034.659962] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14034.670928] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14034.697130] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [14039.836592] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14050.018410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14050.173458] Lustre: tst250fs-OST0001: new disk, initializing [14050.181164] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14050.195671] Lustre: 392887:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14050.329384] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14056.089195] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14058.570258] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14058.585181] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14058.680165] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14061.159378] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14063.862730] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14063.868422] Lustre: Skipped 2 previous similar messages [14084.591766] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14084.610396] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14084.617040] Lustre: Skipped 3 previous similar messages [14084.624132] Lustre: Skipped 2 previous similar messages [14089.710798] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14089.720272] Lustre: Skipped 4 previous similar messages [14091.869179] Lustre: server umount tst250fs-MDT0000 complete [14094.581262] LustreError: 390375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790753399 with bad export cookie 2626970371799334172 [14094.588465] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14094.595425] LustreError: 390375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14094.819625] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14094.836083] Lustre: Skipped 1 previous similar message [14095.017674] Lustre: server umount tst250fs-MDT0001 complete [14101.639689] Lustre: server umount tst250fs-OST0000 complete [14105.661385] Lustre: server umount tst250fs-OST0001 complete [14125.346028] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [14133.490732] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [14163.544629] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:31:06 (1790753466) [14165.016825] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14181.666950] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [14191.668634] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [14227.222702] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13938 sec ========= 03:32:09 (1790753529) [14229.815059] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:32:11 (1790753531) === [14234.160667] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:32:16 (1790753536) === [14277.117940] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [14280.193765] Key type lgssc unregistered [14280.615658] LNet: 402028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14280.627092] LNetError: 402028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14280.644221] LNet: Removed LNI 192.168.203.131@tcp [14281.939273] Key type .llcrypt unregistered [14281.941080] Key type ._llcrypt unregistered