[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468285716 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002270] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009111] LSM: Security Framework initializing [ 0.010037] Yama: becoming mindful. [ 0.011040] SELinux: Initializing. [ 0.012084] *** VALIDATE selinux *** [ 0.020084] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025587] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] *** VALIDATE tmpfs *** [ 0.030484] *** VALIDATE proc *** [ 0.032237] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035052] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042017] debug: unmapping init [mem 0xffffffffaba59000-0xffffffffaba60fff] [ 0.044184] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045810] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046034] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053350] rcu: Hierarchical SRCU implementation. [ 0.055577] smp: Bringing up secondary CPUs ... [ 0.056598] x86: Booting SMP configuration: [ 0.057033] .... node #0, CPUs: #1 #2 #3 [ 0.072174] smp: Brought up 1 node, 4 CPUs [ 0.074017] smpboot: Max logical packages: 1 [ 0.075028] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.215064] node 0 deferred pages initialised in 138ms [ 0.217349] devtmpfs: initialized [ 0.218234] x86/mm: Memory block size: 128MB [ 0.220861] gcov: version magic: 0x41383552 [ 0.222288] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.223102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.224289] pinctrl core: initialized pinctrl subsystem [ 0.225229] [ 0.225815] ************************************************************* [ 0.226012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.227019] ** ** [ 0.228016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.229017] ** ** [ 0.230017] ** This means that this kernel is built to expose internal ** [ 0.231020] ** IOMMU data structures, which may compromise security on ** [ 0.232013] ** your system. ** [ 0.233016] ** ** [ 0.234017] ** If you see this message and you are not debugging the ** [ 0.235015] ** kernel, report this immediately to your vendor! ** [ 0.236013] ** ** [ 0.237019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.238013] ************************************************************* [ 0.239628] NET: Registered protocol family 16 [ 0.240316] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.241072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.242055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.244149] cpuidle: using governor menu [ 0.245687] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.246574] PCI: Using configuration type 1 for base access [ 0.247130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.254085] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.256029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.258104] cryptd: max_cpu_qlen set to 1000 [ 0.261336] ACPI: Added _OSI(Module Device) [ 0.263014] ACPI: Added _OSI(Processor Device) [ 0.264012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.266010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.269926] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.277297] ACPI: Interpreter enabled [ 0.278111] ACPI: PM: (supports S0 S3 S4 S5) [ 0.280028] ACPI: Using IOAPIC for interrupt routing [ 0.282225] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.286634] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.299280] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.302071] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.305031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.308138] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.313569] acpiphp: Slot [2] registered [ 0.316233] acpiphp: Slot [5] registered [ 0.317184] acpiphp: Slot [6] registered [ 0.318170] acpiphp: Slot [7] registered [ 0.320201] acpiphp: Slot [8] registered [ 0.321182] acpiphp: Slot [9] registered [ 0.323176] acpiphp: Slot [10] registered [ 0.325475] acpiphp: Slot [3] registered [ 0.326133] acpiphp: Slot [4] registered [ 0.328129] acpiphp: Slot [11] registered [ 0.329137] acpiphp: Slot [12] registered [ 0.331145] acpiphp: Slot [13] registered [ 0.332157] acpiphp: Slot [14] registered [ 0.334138] acpiphp: Slot [15] registered [ 0.336224] acpiphp: Slot [16] registered [ 0.337140] acpiphp: Slot [17] registered [ 0.339214] acpiphp: Slot [18] registered [ 0.341117] acpiphp: Slot [19] registered [ 0.342107] acpiphp: Slot [20] registered [ 0.344114] acpiphp: Slot [21] registered [ 0.345117] acpiphp: Slot [22] registered [ 0.347173] acpiphp: Slot [23] registered [ 0.349134] acpiphp: Slot [24] registered [ 0.350090] acpiphp: Slot [25] registered [ 0.351082] acpiphp: Slot [26] registered [ 0.352056] acpiphp: Slot [27] registered [ 0.353090] acpiphp: Slot [28] registered [ 0.354227] acpiphp: Slot [29] registered [ 0.356169] acpiphp: Slot [30] registered [ 0.358119] acpiphp: Slot [31] registered [ 0.359107] PCI host bridge to bus 0000:00 [ 0.361023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.363030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.366031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.368024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.371038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.374085] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.376247] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.379272] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.383697] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.393680] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.399564] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.402032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.405045] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.408027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.410703] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.414012] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.417080] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.420944] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.426018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.441036] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.446808] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.453594] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.467027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.482045] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.501032] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.513000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.524022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.533024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.551019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.562565] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.570028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.578121] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.589030] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.607870] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.620024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.633030] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.648015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.666149] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.676016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.679015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.689019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.700607] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.703000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.708016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.723024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.741219] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.744388] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.746352] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.748000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.750203] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.758495] iommu: Default domain type: Passthrough [ 0.759000] SCSI subsystem initialized [ 0.762455] ACPI: bus type USB registered [ 0.765158] usbcore: registered new interface driver usbfs [ 0.767106] usbcore: registered new interface driver hub [ 0.768159] usbcore: registered new device driver usb [ 0.770179] pps_core: LinuxPPS API ver. 1 registered [ 0.771000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.772065] PTP clock support registered [ 0.774362] EDAC MC: Ver: 3.0.0 [ 0.777003] PCI: Using ACPI for IRQ routing [ 0.777813] NetLabel: Initializing [ 0.779011] NetLabel: domain hash size = 128 [ 0.780000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.780000] NetLabel: unlabeled traffic allowed by default [ 0.782372] vgaarb: loaded [ 0.784116] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.786016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.792354] clocksource: Switched to clocksource kvm-clock [ 0.934740] VFS: Disk quotas dquot_6.6.0 [ 0.936650] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.939391] *** VALIDATE ramfs *** [ 0.940973] *** VALIDATE hugetlbfs *** [ 0.943120] pnp: PnP ACPI init [ 0.946634] pnp: PnP ACPI: found 6 devices [ 0.967324] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.971168] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.974244] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.976890] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.979339] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.981827] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.985162] NET: Registered protocol family 2 [ 0.987439] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.993821] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.996768] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.004666] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.010355] TCP: Hash tables configured (established 65536 bind 65536) [ 1.017503] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.022427] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.025944] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.030546] NET: Registered protocol family 1 [ 1.041956] RPC: Registered named UNIX socket transport module. [ 1.049504] RPC: Registered udp transport module. [ 1.054717] RPC: Registered tcp transport module. [ 1.062543] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.067604] NET: Registered protocol family 44 [ 1.070499] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.077690] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.084845] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.097411] pci 0000:00:01.0: quirk_isa_dma_hangs+0x0/0x20 took 12278 usecs [ 1.101210] PCI: CLS 0 bytes, default 64 [ 1.108659] Unpacking initramfs... [ 2.072159] hrtimer: interrupt took 9191809 ns [ 5.019381] debug: unmapping init [mem 0xffff8ec03cc54000-0xffff8ec03ffbffff] [ 5.027334] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.033122] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.040638] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.054138] Initialise system trusted keyrings [ 6.056319] Key type blacklist registered [ 6.060088] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.075331] zbud: loaded [ 6.079605] *** VALIDATE nfs *** [ 6.080920] *** VALIDATE nfs4 *** [ 6.083564] pstore: using deflate compression [ 6.088714] Platform Keyring initialized [ 6.255446] NET: Registered protocol family 38 [ 6.257643] Key type asymmetric registered [ 6.259968] Asymmetric key parser 'x509' registered [ 6.261934] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.272729] io scheduler mq-deadline registered [ 6.275797] io scheduler kyber registered [ 6.278359] io scheduler bfq registered [ 6.284215] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.288721] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.292779] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.295592] ACPI: Power Button [PWRF] [ 6.310407] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.321346] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.337256] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.350890] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.394547] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.428353] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.469440] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.478253] Non-volatile memory driver v1.3 [ 6.481797] Linux agpgart interface v0.103 [ 6.542410] virtio_blk virtio1: [vda] 134840 512-byte logical blocks (69.0 MB/65.8 MiB) [ 6.552780] vda: detected capacity change from 0 to 69038080 [ 6.612883] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.617620] vdb: detected capacity change from 0 to 1073741824 [ 6.680880] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.693360] vdc: detected capacity change from 0 to 2621440000 [ 6.730415] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.733114] vdd: detected capacity change from 0 to 2621440000 [ 6.765635] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.769486] vde: detected capacity change from 0 to 4294967296 [ 6.815140] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.818099] vdf: detected capacity change from 0 to 4294967296 [ 6.828549] libphy: Fixed MDIO Bus: probed [ 6.853145] usbcore: registered new interface driver usbserial_generic [ 6.857332] usbserial: USB Serial support registered for generic [ 6.859355] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.862165] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.863563] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.866152] mousedev: PS/2 mouse device common for all mice [ 6.868886] rtc_cmos 00:05: RTC can wake from S4 [ 6.870853] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.871941] rtc_cmos 00:05: registered as rtc0 [ 6.872034] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.872074] intel_pstate: CPU model not supported [ 6.874829] hid: raw HID events driver (C) Jiri Kosina [ 6.885423] usbcore: registered new interface driver usbhid [ 6.887381] usbhid: USB HID core driver [ 6.888671] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.890424] drop_monitor: Initializing network drop monitor service [ 6.895114] Initializing XFRM netlink socket [ 6.896904] NET: Registered protocol family 10 [ 6.898555] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.900432] Segment Routing with IPv6 [ 6.903875] NET: Registered protocol family 17 [ 6.905979] mpls_gso: MPLS GSO support [ 6.911179] RAS: Correctable Errors collector initialized. [ 6.913286] AVX version of gcm_enc/dec engaged. [ 6.914909] AES CTR mode by8 optimization enabled [ 7.042854] sched_clock: Marking stable (7042752591, 0)->(8009217426, -966464835) [ 7.048434] registered taskstats version 1 [ 7.051462] Loading compiled-in X.509 certificates [ 7.054098] zswap: loaded using pool lzo/zbud [ 7.089792] Key type big_key registered [ 7.105040] Key type encrypted registered [ 7.106795] ima: No TPM chip found, activating TPM-bypass! [ 7.109164] ima: Allocated hash algorithm: sha1 [ 7.111344] ima: No architecture policies found [ 7.113594] evm: Initialising EVM extended attributes: [ 7.115821] evm: security.selinux [ 7.117449] evm: security.ima [ 7.118715] evm: security.capability [ 7.119994] evm: HMAC attrs: 0x1 [ 7.122929] rtc_cmos 00:05: setting system clock to 2026-06-01 18:17:33 UTC (1780337853) [ 7.132769] debug: unmapping init [mem 0xffffffffaca03000-0xffffffffacbfffff] [ 7.139812] debug: unmapping init [mem 0xffffffffab782000-0xffffffffaba58fff] [ 7.150267] Write protecting the kernel read-only data: 28672k [ 7.156272] debug: unmapping init [mem 0xffffffffa9e03000-0xffffffffa9ffffff] [ 7.160292] debug: unmapping init [mem 0xffffffffaa714000-0xffffffffaa7fffff] [ 7.240496] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.270607] systemd[1]: Detected virtualization kvm. [ 7.277188] systemd[1]: Detected architecture x86-64. [ 7.282125] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.319244] systemd[1]: No hostname configured. [ 7.323152] systemd[1]: Set hostname to . [ 7.325081] random: systemd: uninitialized urandom read (16 bytes read) [ 7.329793] systemd[1]: Initializing machine ID from random generator. [ 7.711740] random: systemd: uninitialized urandom read (16 bytes read) [ 7.714249] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 7.720406] random: systemd: uninitialized urandom read (16 bytes read) [ 7.724561] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 7.736923] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.413879] device-mapper: uevent: version 1.0.3 [ 9.419944] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 11.063129] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.138337] random: fast init done [ 11.315856] scsi host0: ata_piix [ 11.331206] scsi host1: ata_piix [ 11.332800] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.334850] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.636882] random: crng init done [ 16.639120] random: 7 urandom warning(s) missed due to ratelimiting [ 18.674841] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ 20.286688] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.090203] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.901516] SELinux: Disabled at runtime. [ 24.056285] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.070937] systemd[1]: Detected virtualization kvm. [ 24.074158] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.819141] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.823165] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.834858] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.842364] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.850186] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.868462] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.875644] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ 26.072163] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 27.402612] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.459028] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.512573] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.088559] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.169737] EDAC sbridge: Ver: 1.1.2 [ 32.443818] Key type dns_resolver registered [ 33.064276] NFS: Registering the id_resolver key type [ 33.071354] Key type id_resolver registered [ 33.074277] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg105-server login: [ 91.702453] libcfs: loading out-of-tree module taints kernel. [ 91.726365] Key type ._llcrypt registered [ 91.728172] Key type .llcrypt registered [ 91.822446] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 111.727786] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 113.267885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 113.304024] alg: No test for adler32 (adler32-zlib) [ 115.078516] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 116.431846] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 118.439272] Key type lgssc registered [ 120.854218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 138.359334] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 178.512906] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 190.852507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 190.881129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.240888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 192.268755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 192.323961] Lustre: lustre-MDT0000: new disk, initializing [ 192.403150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.422881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.327936] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.514155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.608105] Lustre: 6507:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 209.647392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 209.656856] Lustre: Skipped 1 previous similar message [ 209.745303] Lustre: lustre-MDT0001: new disk, initializing [ 209.813846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.836377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 209.859222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.349651] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.079969] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 231.783135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.080897] Lustre: lustre-OST0000: new disk, initializing [ 232.084985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 232.092362] Lustre: 8413:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.187418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 233.114579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 233.131312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 233.183239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.941044] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.682177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.825301] Lustre: lustre-OST0001: new disk, initializing [ 251.828574] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.835177] Lustre: 9470:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.927825] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.591694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 257.608278] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 257.696312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 258.228514] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.206480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 275.399446] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 283.130720] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing check_logdir /tmp/testlogs/ [ 289.475841] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing yml_node [ 295.272489] Lustre: DEBUG MARKER: Client: 2.17.53.28 [ 298.024128] Lustre: DEBUG MARKER: MDS: 2.17.53.28 [ 300.675972] Lustre: DEBUG MARKER: OSS: 2.17.53.28 [ 302.433390] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jun 1 14:22:26 EDT 2026 [ 320.642212] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 322.334350] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 337.888182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 337.906271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.929780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.936394] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.938962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.954560] Lustre: Skipped 2 previous similar messages [ 339.970743] Lustre: Skipped 3 previous similar messages [ 345.057587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.066832] Lustre: Skipped 3 previous similar messages [ 350.175884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.189870] Lustre: Skipped 3 previous similar messages [ 352.223492] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 352.524192] Lustre: server umount lustre-MDT0000 complete [ 357.198838] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338203 with bad export cookie 997732489930147851 [ 357.199101] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.204434] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 360.425311] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.433239] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.440731] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.461615] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 360.480019] Lustre: Skipped 1 previous similar message [ 362.463672] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.486993] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 363.845548] Lustre: server umount lustre-MDT0001 complete [ 367.980503] Lustre: server umount lustre-OST0000 complete [ 373.013901] Lustre: server umount lustre-OST0001 complete [ 381.714543] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 397.749786] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 436.477661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.709847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 436.733862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 436.786405] Lustre: lustre-MDT0000: new disk, initializing [ 436.846703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.874549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 440.296667] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 450.488990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.589667] Lustre: 15005:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 450.649862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 450.653983] Lustre: Skipped 1 previous similar message [ 450.732281] Lustre: lustre-MDT0001: new disk, initializing [ 450.898785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 450.959972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 450.968660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 455.577392] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 463.964899] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 467.955421] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 474.812637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 475.216637] Lustre: lustre-OST0000: new disk, initializing [ 475.220060] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 475.227661] Lustre: 16746:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 475.376359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.848597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 476.856891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 476.988212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 482.720202] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 491.560635] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 497.344500] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 497.582158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 502.787300] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 503.105188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 507.872641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 507.880606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.888869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.891014] Lustre: Skipped 4 previous similar messages [ 511.564404] Lustre: server umount lustre-OST0000 complete [ 517.602125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 517.632167] LustreError: Skipped 1 previous similar message [ 517.654285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.695689] Lustre: Skipped 1 previous similar message [ 522.169601] Lustre: server umount lustre-MDT0000 complete [ 523.234214] LustreError: 15013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.246215] LustreError: 15013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 526.456717] LustreError: 14996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338372 with bad export cookie 997732489930149384 [ 526.464619] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 526.467151] LustreError: 14996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 526.897760] Lustre: server umount lustre-MDT0001 complete [ 533.900596] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:26:18 (1780338378) [ 554.460826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.909048] LustreError: 19474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.986662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.726637] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 560.096174] LustreError: 19475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.138613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.340221] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 570.343020] Lustre: Skipped 3 previous similar messages [ 576.441404] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 583.129594] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 586.643582] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 589.160312] Lustre: Failing over lustre-MDT0000 [ 589.374473] Lustre: server umount lustre-MDT0000 complete [ 590.304629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.329108] Lustre: Skipped 1 previous similar message [ 590.336489] LustreError: 19471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.357507] LustreError: 19471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 605.666839] Lustre: 3661:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780338436/real 1780338436] req@ffff8ec09fe4fb80 x1866819659001856/t0(0) o400->MGC192.168.201.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780338452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 605.683379] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.035446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.790857] LustreError: 19469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.805072] LustreError: 19469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 615.908530] LustreError: 3657:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ebf82dac380 x1866819659004672/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 616.241674] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 620.934883] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 621.536211] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 621.547493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 621.558689] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 628.325829] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 635.886346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.891841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 635.916884] Lustre: Skipped 2 previous similar messages [ 635.930151] Lustre: Skipped 1 previous similar message [ 638.709857] Lustre: server umount lustre-MDT0000 complete [ 642.411133] LustreError: 19456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338488 with bad export cookie 997732489930150798 [ 642.419197] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 642.427929] LustreError: 19456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 642.731608] Lustre: server umount lustre-MDT0001 complete [ 651.003440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.442390] LustreError: 22471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.462232] LustreError: 22471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 651.523393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.529587] Lustre: Skipped 2 previous similar messages [ 655.927571] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 664.806771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.940247] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 675.545483] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 679.522162] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 682.040590] Lustre: Failing over lustre-MDT0000 [ 682.530449] Lustre: server umount lustre-MDT0000 complete [ 685.540127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 685.550093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.433250] Lustre: 3658:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780338532/real 1780338532] req@ffff8ebf82c0b800 x1866819659031296/t0(0) o400->MGC192.168.201.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780338548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 702.455708] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.495795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.649878] LustreError: 3657:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ebf82ebc700 x1866819659034112/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 712.122474] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 716.989550] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 717.285194] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 717.287206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 717.304543] Lustre: Skipped 1 previous similar message [ 717.331774] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 724.664397] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 732.647596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 743.391151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 743.767406] Lustre: server umount lustre-MDT0000 complete [ 745.982928] LustreError: 24717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 746.004106] LustreError: 24717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 748.478477] LustreError: 22451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338594 with bad export cookie 997732489930151736 [ 748.508289] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 748.777042] Lustre: server umount lustre-MDT0001 complete [ 758.270271] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:30:03 (1780338603) [ 776.380739] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 787.640737] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 838.277605] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 851.075504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.465955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 851.504391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 851.585380] Lustre: lustre-MDT0000: new disk, initializing [ 851.768021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 851.781163] Lustre: Skipped 2 previous similar messages [ 851.831805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 856.328052] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 867.294661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.421713] Lustre: 28455:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 867.745384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 873.294690] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 882.799951] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 887.130824] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 896.023965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 896.335234] Lustre: 30196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 897.652276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 897.699917] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 897.797707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 897.835972] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 904.430310] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 913.182565] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 918.736297] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 944.108480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 944.111530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.113226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 944.113234] Lustre: Skipped 5 previous similar messages [ 944.145168] Lustre: Skipped 4 previous similar messages [ 954.335634] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 954.620544] Lustre: server umount lustre-OST0000 complete [ 965.370979] Lustre: server umount lustre-MDT0000 complete [ 969.517477] LustreError: 30195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338815 with bad export cookie 997732489930152142 [ 969.532169] LustreError: 30195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 969.539873] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 970.184439] Lustre: server umount lustre-MDT0001 complete [ 983.907583] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 988.073471] Key type lgssc unregistered [ 988.506503] LNet: 32050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.527391] LNetError: 32050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.567184] LNet: Removed LNI 192.168.201.105@tcp [ 989.905157] Key type .llcrypt unregistered [ 989.907051] Key type ._llcrypt unregistered [ 1019.601394] Key type ._llcrypt registered [ 1019.604408] Key type .llcrypt registered [ 1019.722576] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1021.047151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1021.212851] alg: No test for adler32 (adler32-zlib) [ 1022.490077] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 1022.781148] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1024.607229] Key type lgssc registered [ 1025.924526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1033.960355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1033.971049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.600305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1042.295761] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1044.968913] LustreError: 33448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.005125] LustreError: 33448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1050.138978] LustreError: 33449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.645910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.258173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1060.135547] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1066.365620] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1070.465298] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1078.503605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.816234] LustreError: 35044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.244811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.261448] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1085.315768] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1092.040596] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1108.628856] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:35:52 (1780338952) [ 1110.754851] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1113.094954] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1120.738580] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:36:05 (1780338965) [ 1131.154972] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:36:15 (1780338975) [ 1132.316958] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1142.865579] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1142.875484] Lustre: Skipped 2 previous similar messages [ 1145.202489] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:36:29 (1780338989) [ 1147.532295] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1147.536954] Lustre: Skipped 2 previous similar messages [ 1156.807367] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1156.812420] Lustre: Skipped 4 previous similar messages [ 1173.018641] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:36:57 (1780339017) [ 1176.312435] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1181.456492] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1191.421268] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:37:15 (1780339035) [ 1193.518853] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1193.522558] Lustre: Skipped 6 previous similar messages [ 1210.772352] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:37:34 (1780339054) [ 1228.770951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1228.774610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.779534] LustreError: Skipped 1 previous similar message [ 1228.781663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1228.790321] Lustre: Skipped 1 previous similar message [ 1228.803104] Lustre: Skipped 1 previous similar message [ 1233.889730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1233.915437] Lustre: Skipped 1 previous similar message [ 1239.011569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1239.019242] Lustre: Skipped 1 previous similar message [ 1240.031292] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1240.297996] Lustre: server umount lustre-OST0000 complete [ 1243.620971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1243.630236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.648980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1249.251639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1249.259116] Lustre: Skipped 3 previous similar messages [ 1249.982429] Lustre: server umount lustre-MDT0000 complete [ 1253.514960] LustreError: 36030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339099 with bad export cookie 15716388327409683001 [ 1253.515233] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1253.540877] LustreError: 36030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1253.863144] Lustre: server umount lustre-MDT0001 complete [ 1266.573577] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1269.420318] Key type lgssc unregistered [ 1269.659601] LNet: 40962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.666805] LNetError: 40962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.676597] LNet: Removed LNI 192.168.201.105@tcp [ 1270.352910] Key type .llcrypt unregistered [ 1270.354620] Key type ._llcrypt unregistered [ 1289.399732] Key type ._llcrypt registered [ 1289.402295] Key type .llcrypt registered [ 1289.536931] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 1306.399892] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1307.853328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1307.880253] alg: No test for adler32 (adler32-zlib) [ 1309.066231] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 1309.462049] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1311.231107] Key type lgssc registered [ 1312.885846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1350.769782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1350.814744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.078729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1352.115973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1352.213733] Lustre: lustre-MDT0000: new disk, initializing [ 1352.409149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.440660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1357.588071] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1369.734329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.900580] Lustre: 44083:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1369.954964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1369.961646] Lustre: Skipped 1 previous similar message [ 1370.103355] Lustre: lustre-MDT0001: new disk, initializing [ 1370.262326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1370.306955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1370.323140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1375.226405] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1383.703760] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1388.401971] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1396.043153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1396.337982] Lustre: lustre-OST0000: new disk, initializing [ 1396.342767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1396.355286] Lustre: 45828:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1396.479988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.640387] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1406.507963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1406.516914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1406.751682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1411.030301] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1415.559977] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1415.915675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1421.618742] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1421.855749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1427.433634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1427.433992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.467140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1429.985733] Lustre: server umount lustre-OST0000 complete [ 1436.647758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1436.663771] LustreError: Skipped 1 previous similar message [ 1436.670727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.685612] Lustre: Skipped 1 previous similar message [ 1436.691793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1436.697182] Lustre: Skipped 1 previous similar message [ 1440.748985] Lustre: server umount lustre-MDT0000 complete [ 1445.165846] LustreError: 44077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339291 with bad export cookie 975344489195913386 [ 1445.167918] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1445.177390] LustreError: 44077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1445.722261] Lustre: server umount lustre-MDT0001 complete [ 1456.162566] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:41:40 (1780339300) [ 1463.590740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.995323] LustreError: 48140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.012981] LustreError: 48140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1464.084549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.207648] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1469.413584] LustreError: 48140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.456716] LustreError: 48140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1474.539111] LustreError: 48141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.674848] LustreError: 48140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1481.888062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.349695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1486.686387] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1496.067131] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1501.748594] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1512.119979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1512.560932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1520.003199] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1526.621599] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1543.086226] LustreError: 50353:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1543.995308] Lustre: Modifying parameter general.timeout=1 in log params [ 1545.010133] Lustre: Modifying parameter general.timeout=2 in log params [ 1547.120288] Lustre: Modifying parameter general.timeout=4 in log params [ 1547.125334] Lustre: Skipped 1 previous similar message [ 1551.420926] Lustre: Modifying parameter general.timeout=8 in log params [ 1551.427302] Lustre: Skipped 3 previous similar messages [ 1560.023589] Lustre: Modifying parameter general.timeout=17 in log params [ 1560.028384] Lustre: Skipped 8 previous similar messages [ 1576.966182] Lustre: Modifying parameter general.timeout=34 in log params [ 1576.974361] Lustre: Skipped 16 previous similar messages [ 1609.601196] Lustre: Modifying parameter general.timeout=68 in log params [ 1609.607773] Lustre: Skipped 33 previous similar messages [ 1674.605651] Lustre: Modifying parameter general.timeout=129 in log params [ 1674.612970] Lustre: Skipped 60 previous similar messages [ 1688.569610] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:45:32 (1780339532) [ 1692.403777] Lustre: 57306:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1694.387085] Lustre: 57417:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1696.764371] Lustre: 57515:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1701.520494] Lustre: 57722:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1701.531381] Lustre: 57722:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1733.891765] Lustre: 59181:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1733.905516] Lustre: 59181:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1733.928101] LustreError: 59181:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1738.546254] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:46:22 (1780339582) [ 1753.570667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1753.579541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.590364] Lustre: Skipped 1 previous similar message [ 1753.600253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.602790] Lustre: Skipped 2 previous similar messages [ 1759.142282] Lustre: server umount lustre-MDT0000 complete [ 1759.211634] LustreError: 48136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1759.234818] LustreError: 48136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1763.708688] LustreError: 48121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339610 with bad export cookie 975344489195914268 [ 1763.712732] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.721341] LustreError: 48121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1764.013372] Lustre: server umount lustre-MDT0001 complete [ 1778.913872] Lustre: server umount lustre-OST0000 complete [ 1807.782517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1808.231736] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1808.308197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1808.323587] Lustre: Skipped 23 previous similar messages [ 1808.644479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1813.428733] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1822.580517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.848558] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1822.898823] Lustre: 61717:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1827.914807] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1833.939653] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1838.158862] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1846.098103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1846.309572] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1846.627777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.634505] Lustre: Skipped 1 previous similar message [ 1852.678177] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1859.955223] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1862.024277] Lustre: 63942:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1873.889708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1873.910162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.930951] Lustre: Skipped 2 previous similar messages [ 1873.939266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1873.948949] Lustre: Skipped 3 previous similar messages [ 1877.713560] Lustre: server umount lustre-OST0000 complete [ 1884.645576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1884.652385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.661850] LustreError: Skipped 1 previous similar message [ 1884.665280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.665292] Lustre: Skipped 2 previous similar messages [ 1884.719073] Lustre: Skipped 2 previous similar messages [ 1888.761491] Lustre: server umount lustre-MDT0000 complete [ 1889.771296] LustreError: 61728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.787857] LustreError: 61728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1892.827223] LustreError: 62441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339739 with bad export cookie 975344489195917453 [ 1892.835619] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1892.843135] LustreError: 62441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1893.178986] Lustre: server umount lustre-MDT0001 complete [ 1905.740887] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1909.240065] Key type lgssc unregistered [ 1909.607607] LNet: 65134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.613638] LNetError: 65134:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.635277] LNet: Removed LNI 192.168.201.105@tcp [ 1910.749719] Key type .llcrypt unregistered [ 1910.751211] Key type ._llcrypt unregistered [ 1936.345940] Key type ._llcrypt registered [ 1936.350585] Key type .llcrypt registered [ 1936.492942] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1937.868841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1937.951123] alg: No test for adler32 (adler32-zlib) [ 1939.209941] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 1939.485591] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1941.223516] Key type lgssc registered [ 1942.963086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1951.211383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1951.231102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.984237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.784383] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1963.489989] LustreError: 66530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1963.515493] LustreError: 66530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1968.181388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.697742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1973.644585] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1981.089193] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1986.031219] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1993.303115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1993.951367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.108271] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2007.004840] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2018.048156] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2029.026966] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.039877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2029.055034] Lustre: Skipped 1 previous similar message [ 2029.070533] Lustre: Skipped 2 previous similar messages [ 2034.146473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2034.692115] Lustre: server umount lustre-MDT0000 complete [ 2039.109269] LustreError: 68878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339885 with bad export cookie 4338725324101936860 [ 2039.111995] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.123816] LustreError: 68878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2039.271282] LustreError: 66524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.272879] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.300891] Lustre: Skipped 1 previous similar message [ 2039.457600] Lustre: server umount lustre-MDT0001 complete [ 2046.754116] Lustre: server umount lustre-OST0000 complete [ 2080.415828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.830980] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2080.890335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2081.314147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.876248] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2096.898798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.026814] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2097.071584] Lustre: 71068:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2097.178455] LustreError: 71078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.190985] LustreError: 71078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2097.674274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2103.784128] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2111.257097] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2116.168944] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2124.344476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2124.657656] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2125.102072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.078222] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2140.651548] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2142.590211] Lustre: 73304:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2144.985148] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2154.981514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2154.983290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.995420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2154.995431] Lustre: Skipped 2 previous similar messages [ 2155.000699] LustreError: Skipped 1 previous similar message [ 2155.049774] Lustre: Skipped 1 previous similar message [ 2156.574558] Lustre: server umount lustre-OST0000 complete [ 2164.218114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2164.244529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.274194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.292428] Lustre: Skipped 1 previous similar message [ 2167.826189] Lustre: server umount lustre-MDT0000 complete [ 2170.342742] LustreError: 71078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.374852] LustreError: 71078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2172.322931] LustreError: 71059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340018 with bad export cookie 4338725324101937707 [ 2172.328698] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2172.333482] LustreError: 71059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2172.729920] Lustre: server umount lustre-MDT0001 complete [ 2186.300832] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2189.440747] Key type lgssc unregistered [ 2189.870478] LNet: 74499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.877523] LNetError: 74499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2190.957099] LNet: Removed LNI 192.168.201.105@tcp [ 2192.349893] Key type .llcrypt unregistered [ 2192.351533] Key type ._llcrypt unregistered [ 2223.239065] Key type ._llcrypt registered [ 2223.241457] Key type .llcrypt registered [ 2223.452686] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2224.565320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.724121] alg: No test for adler32 (adler32-zlib) [ 2225.885911] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 2226.171125] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2228.023238] Key type lgssc registered [ 2229.801970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2238.606611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2238.620146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.252401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.922923] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2249.699302] LustreError: 75899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.725456] LustreError: 75899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2254.818719] LustreError: 75900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.322585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.803358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2260.171315] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2265.897709] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2270.674928] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2278.081107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2278.967396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.096209] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2292.205144] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2300.099632] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2301.154545] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2302.359534] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2304.618733] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2304.627446] Lustre: Skipped 1 previous similar message [ 2308.953181] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2308.957961] Lustre: Skipped 3 previous similar messages [ 2317.231887] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2317.236871] Lustre: Skipped 7 previous similar messages [ 2333.709903] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2333.713516] Lustre: Skipped 13 previous similar messages [ 2365.918723] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2365.925806] Lustre: Skipped 27 previous similar messages [ 2421.342246] Lustre: 83673:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2424.465666] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:57:48 (1780340268) [ 2426.638980] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2429.384754] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:57:53 (1780340273) [ 2438.624540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2438.641315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.661782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2439.146044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2439.147404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.181179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2439.183538] Lustre: Skipped 1 previous similar message [ 2440.386183] Lustre: server umount lustre-OST0000 complete [ 2445.281978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2445.293455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.311543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2449.381266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2449.388068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.698517] Lustre: server umount lustre-MDT0000 complete [ 2454.501343] LustreError: 77522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.533388] LustreError: 77522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2456.312039] LustreError: 75880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340302 with bad export cookie 14938136238927383591 [ 2456.320021] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2456.340503] LustreError: 75880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2456.813385] Lustre: server umount lustre-MDT0001 complete [ 2469.882827] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2473.117834] Key type lgssc unregistered [ 2473.589845] LNet: 84906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.604059] LNetError: 84906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2473.631335] LNet: Removed LNI 192.168.201.105@tcp [ 2474.459094] Key type .llcrypt unregistered [ 2474.460702] Key type ._llcrypt unregistered [ 2479.304590] Key type ._llcrypt registered [ 2479.309917] Key type .llcrypt registered [ 2479.547775] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2485.246782] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules [ 2486.044707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2486.141083] alg: No test for adler32 (adler32-zlib) [ 2487.362306] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 2487.796229] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2489.703287] Key type lgssc registered [ 2491.324683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2492.722152] LustreError: 86088:0:(lu_object.c:1479:lu_context_key_register()) cfs_race id 60d sleeping [ 2514.032854] LustreError: 86088:0:(lu_object.c:1479:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2514.722871] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2525.196040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2525.250207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.090745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2531.260906] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2537.445155] LustreError: 87080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.467612] LustreError: 87080:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2541.773681] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:59:45 (1780340385) [ 2542.560581] LustreError: 87081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.861765] LustreError: 87112:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2546.874424] LustreError: 87112:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2546.881795] LustreError: 87112:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2547.362067] Lustre: server umount lustre-MDT0000 complete [ 2564.662707] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2568.487711] Key type lgssc unregistered [ 2568.783849] LNet: 88575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2568.800362] LNetError: 88575:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2568.828457] LNet: Removed LNI 192.168.201.105@tcp [ 2569.769696] Key type .llcrypt unregistered [ 2569.772949] Key type ._llcrypt unregistered [ 2590.706330] Key type ._llcrypt registered [ 2590.713526] Key type .llcrypt registered [ 2590.944869] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2592.218079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2592.299966] alg: No test for adler32 (adler32-zlib) [ 2593.630128] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 2594.235946] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2596.119247] Key type lgssc registered [ 2597.888859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2607.388688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2607.405973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.261641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2615.047437] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2618.855323] LustreError: 89778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.878608] LustreError: 89778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2623.988319] LustreError: 89777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.056355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.876707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2633.278273] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2640.011731] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2645.373098] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2654.458527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.144600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.955973] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2669.377502] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2734.845744] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:02:59 (1780340579) [ 2736.872913] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2738.801867] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:03:03 (1780340583) [ 2745.313483] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2745.328822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2745.358781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2750.441201] LustreError: 89774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.462017] LustreError: 89774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2750.590260] Lustre: server umount lustre-MDT0000 complete [ 2755.479693] LustreError: 89758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340601 with bad export cookie 7658560667975268964 [ 2755.485874] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2755.491769] LustreError: 89758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2755.574177] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.580056] LustreError: 89773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.588720] Lustre: Skipped 2 previous similar messages [ 2755.589408] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2755.589411] Lustre: Skipped 2 previous similar messages [ 2756.188259] Lustre: server umount lustre-MDT0001 complete [ 2761.878788] Lustre: server umount lustre-OST0000 complete [ 2779.909642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.373261] LustreError: 93768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.592137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.411257] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2789.866511] LustreError: 93769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2789.892422] LustreError: 93769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2795.105284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.921430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2800.555444] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2806.610566] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2810.644260] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2825.067026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2825.294611] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2825.299140] LustreError: 93756:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2825.312950] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2825.322146] LustreError: 95624:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2825.326786] LustreError: 95624:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2825.333237] LustreError: 95624:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2825.451180] Lustre: server umount lustre-OST0000 complete [ 2825.457340] LustreError: 95624:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2833.721427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.918748] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2833.923625] LustreError: 93758:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2833.943845] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2833.948662] LustreError: 96090:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2833.952624] LustreError: 96090:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2833.955783] LustreError: 96090:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2834.019963] Lustre: server umount lustre-OST0000 complete [ 2834.024233] LustreError: 96090:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2844.008174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2844.243207] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2844.245550] Lustre: Found index 0 for lustre-OST0000, updating log [ 2844.258176] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2844.407285] Lustre: lustre-OST0000: new disk, initializing [ 2844.411608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2844.419896] Lustre: 96625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2844.883990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.674746] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2858.557948] Lustre: Failing over lustre-OST0000 [ 2858.815286] Lustre: server umount lustre-OST0000 complete [ 2865.509993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2865.533912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2865.542777] Lustre: Skipped 1 previous similar message [ 2870.753965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.770582] Lustre: Skipped 1 previous similar message [ 2872.954426] Lustre: server umount lustre-MDT0000 complete [ 2875.882536] LustreError: 97057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.904503] LustreError: 97057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2878.163926] LustreError: 93748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340724 with bad export cookie 7658560667975271701 [ 2878.167775] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2878.186057] LustreError: 93748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2878.695858] Lustre: server umount lustre-MDT0001 complete [ 2889.282656] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:05:33 (1780340733) [ 2908.123602] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2919.566955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.152305] LustreError: 98942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.282062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.078566] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2938.518705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.657749] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2947.156170] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2954.290023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2954.982154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2954.987472] Lustre: Skipped 1 previous similar message [ 2954.996888] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2961.236062] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2964.220876] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2964.224376] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2964.233454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2965.297763] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:09 [ 2965.323495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2965.336295] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2965.338136] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2965.360070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2970.654814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2970.839713] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2970.998248] Lustre: lustre-OST0001: new disk, initializing [ 2971.005843] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2971.026961] Lustre: 101210:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2979.646842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2979.679902] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2979.827638] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2980.124208] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2991.778925] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2998.738412] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3004.117983] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 3005.418446] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.427691] Lustre: Skipped 2 previous similar messages [ 3008.852444] Lustre: server umount lustre-MDT0001 complete [ 3009.160247] LustreError: 98939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.180463] LustreError: 98939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3019.072709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.252102] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3019.261081] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3019.320382] Lustre: 98930:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3019.898148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3019.904079] Lustre: Skipped 1 previous similar message [ 3020.488894] LustreError: 102738:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3020.509507] LustreError: 102738:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3020.515990] LustreError: 98934:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3020.594562] LustreError: 102750:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.105@tcp: cfg command failed: rc = -17 [ 3020.600529] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3020.600529] [ 3024.527964] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3025.412113] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3036.546122] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 15:08:00 (1780340880) [ 3056.114465] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3076.723642] Lustre: 104778:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3102.221069] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3107.096587] Lustre: 105913:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3193.826273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3193.847729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.862825] Lustre: Skipped 1 previous similar message [ 3193.871879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3193.877035] Lustre: Skipped 3 previous similar messages [ 3199.264517] Lustre: server umount lustre-MDT0000 complete [ 3199.457777] LustreError: 99914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.466192] LustreError: 99914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3203.354736] LustreError: 98922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341049 with bad export cookie 7658560667975272709 [ 3203.356901] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3203.365556] LustreError: 98922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3203.954388] Lustre: server umount lustre-MDT0001 complete [ 3217.926628] Lustre: server umount lustre-OST0000 complete [ 3219.745112] Lustre: 89089:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780341050/real 1780341050] req@ffff8ebf8d77f100 x1866822257579648/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780341066 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3219.765191] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.770484] Lustre: Skipped 3 previous similar messages [ 3223.334700] Lustre: server umount lustre-OST0001 complete [ 3230.845703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3245.299906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3258.615391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3270.299104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3293.031629] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3305.387447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.399701] Lustre: lustre-MDT0000: reset Object Index mappings [ 3305.926418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.975844] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3321.661950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.724563] Lustre: lustre-MDT0001: reset Object Index mappings [ 3328.966501] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3332.294392] Lustre: 110974:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3339.255769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3347.519669] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3348.443570] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3351.627552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3358.319657] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3364.360748] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3364.365172] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3365.255100] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3374.966367] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3379.675875] Lustre: 112818:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3432.716606] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 15:14:36 (1780341276) [ 3460.586765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3460.594136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.614485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3460.627677] Lustre: Skipped 4 previous similar messages [ 3465.687823] Lustre: server umount lustre-MDT0000 complete [ 3466.750718] LustreError: 111536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3466.776987] LustreError: 111536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 3469.783278] LustreError: 110976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341316 with bad export cookie 7658560667975319784 [ 3469.789385] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3469.793321] LustreError: 110976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3476.417537] Lustre: server umount lustre-MDT0001 complete [ 3486.762760] Lustre: server umount lustre-OST0000 complete [ 3496.903831] Lustre: server umount lustre-OST0001 complete [ 3503.135611] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3514.839845] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3549.249591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.495554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3549.525267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3549.591898] Lustre: lustre-MDT0000: new disk, initializing [ 3549.659447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3553.703832] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3561.485441] Lustre: server umount lustre-MDT0000 complete [ 3569.793822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.089586] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3570.104749] Lustre: 117037:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3570.120612] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3570.328271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.335775] Lustre: Skipped 4 previous similar messages [ 3574.254634] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3579.047747] Lustre: server umount lustre-MDT0000 complete [ 3586.325169] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:17:11 (1780341431) [ 3587.766739] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3589.463801] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:17:14 (1780341434) [ 3604.924635] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3613.894824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.844145] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3625.908174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.041900] Lustre: 119053:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3626.099197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3626.102013] Lustre: Skipped 1 previous similar message [ 3626.159231] Lustre: lustre-MDT0001: new disk, initializing [ 3626.274081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3626.286344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3629.973631] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3634.410946] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3639.855469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3640.356520] Lustre: lustre-OST0000: new disk, initializing [ 3640.359503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3640.369949] Lustre: 120594:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3642.227675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3642.234283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3642.316686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3646.108780] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3655.052333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3655.199997] Lustre: lustre-OST0001: new disk, initializing [ 3655.203243] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3655.208630] Lustre: 121449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3656.356015] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3656.362022] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3656.418719] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3660.311800] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3668.922610] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3672.274672] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3972.429504] Lustre: 127835:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3972.437390] Lustre: 127835:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3972.443748] Lustre: 127835:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3972.457308] Lustre: 127835:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3972.464864] Lustre: 127835:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3972.470981] Lustre: 127835:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4141.997624] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:26:26 (1780341986) [ 4156.478774] Lustre: server umount lustre-MDT0000 complete [ 4158.273194] LustreError: 119045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342004 with bad export cookie 7658560667975387943 [ 4158.278139] LustreError: 119045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4158.283019] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4158.433703] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4158.436319] LustreError: 122255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4158.437924] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4158.437930] Lustre: Skipped 8 previous similar messages [ 4158.439077] Lustre: Skipped 7 previous similar messages [ 4158.446263] LustreError: 122255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4160.515090] Lustre: server umount lustre-MDT0001 complete [ 4168.668851] Lustre: server umount lustre-OST0000 complete [ 4177.117418] Lustre: server umount lustre-OST0001 complete [ 4181.178189] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 4186.979545] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4207.285576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.453415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4207.475511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4207.524315] Lustre: lustre-MDT0000: new disk, initializing [ 4207.573392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.575546] Lustre: Skipped 4 previous similar messages [ 4207.584375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4210.280630] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4216.977596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4217.058469] Lustre: 130833:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4217.267710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4219.805830] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4224.351629] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4226.312213] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4229.695367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4229.867329] Lustre: 132576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4231.178383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4231.182022] Lustre: Skipped 1 previous similar message [ 4231.183839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4231.238824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4233.083317] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4237.747923] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4242.802937] Lustre: lustre-MDD0000: changelog on [ 4244.419879] Lustre: lustre-MDD0001: changelog on [ 5233.044959] Lustre: lustre-MDD0001: changelog off [ 5234.060317] Lustre: 130839:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5234.810859] Lustre: lustre-MDD0000: changelog off [ 5239.776063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5239.776989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5239.785318] LustreError: Skipped 1 previous similar message [ 5239.788204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5239.797594] Lustre: Skipped 1 previous similar message [ 5239.810195] Lustre: Skipped 2 previous similar messages [ 5244.900577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5244.905554] Lustre: Skipped 1 previous similar message [ 5245.353514] Lustre: server umount lustre-OST0000 complete [ 5250.016516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5250.021534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5250.027382] Lustre: Skipped 1 previous similar message [ 5250.032031] Lustre: Skipped 1 previous similar message [ 5253.356993] Lustre: server umount lustre-MDT0000 complete [ 5255.136625] LustreError: 132075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5255.145106] LustreError: 132075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5255.474980] LustreError: 130825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343101 with bad export cookie 7658560667976200923 [ 5255.477286] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5255.481499] LustreError: 130825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5255.679981] Lustre: server umount lustre-MDT0001 complete [ 5262.305365] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5264.023061] Key type lgssc unregistered [ 5264.242526] LNet: 141433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5264.249817] LNetError: 141433:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5264.262352] LNet: Removed LNI 192.168.201.105@tcp [ 5264.882162] Key type .llcrypt unregistered [ 5264.885114] Key type ._llcrypt unregistered [ 5276.110626] Key type ._llcrypt registered [ 5276.111836] Key type .llcrypt registered [ 5276.167975] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5284.994555] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5285.742114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5285.770859] alg: No test for adler32 (adler32-zlib) [ 5286.771320] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 5286.938330] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5288.575176] Key type lgssc registered [ 5289.383311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5310.131242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5310.148788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5311.301379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5311.319112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5311.365698] Lustre: lustre-MDT0000: new disk, initializing [ 5311.417807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5311.432254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5313.855546] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5320.622650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.707457] Lustre: 144588:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5320.749717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5320.752705] Lustre: Skipped 1 previous similar message [ 5320.801893] Lustre: lustre-MDT0001: new disk, initializing [ 5320.861036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5320.875970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5320.878987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5323.311132] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5327.932943] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5330.130761] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5334.450663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5334.618337] Lustre: lustre-OST0000: new disk, initializing [ 5334.621289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5334.624827] Lustre: 146330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5334.672973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.026953] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5340.713507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5340.725438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5340.803254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5342.816368] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5347.088073] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_module obdecho/obdecho [ 5348.343745] LustreError: 147201:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5348.349330] LustreError: 147201:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5349.442458] LustreError: 147297:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 5356.514251] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5356.516200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5356.521392] Lustre: Skipped 1 previous similar message [ 5356.536647] Lustre: Skipped 2 previous similar messages [ 5361.632095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5361.636946] Lustre: Skipped 2 previous similar messages [ 5362.097953] Lustre: server umount lustre-MDT0000 complete [ 5364.285676] LustreError: 144581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343210 with bad export cookie 3713470657117937145 [ 5364.290125] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5364.300602] LustreError: 144581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5364.476533] Lustre: server umount lustre-MDT0001 complete [ 5376.699948] Lustre: server umount lustre-OST0000 complete [ 5388.934681] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5396.083125] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5413.519131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5413.632202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5413.657373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5413.734959] Lustre: lustre-MDT0000: new disk, initializing [ 5413.783732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5413.793405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5415.890147] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5421.653556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5421.698775] Lustre: 150971:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5421.722993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5421.726035] Lustre: Skipped 1 previous similar message [ 5421.763959] Lustre: lustre-MDT0001: new disk, initializing [ 5421.799861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5421.818787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5421.824234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5423.671969] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5428.058869] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5430.149157] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5434.111534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5434.262523] Lustre: lustre-OST0000: new disk, initializing [ 5434.264922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5434.269015] Lustre: 152710:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5434.326477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5437.094782] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5440.522773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5440.525995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5440.570593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5440.995973] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5443.526631] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5443.686652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5446.028611] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5446.174635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5450.723513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.725293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.729070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5450.733271] Lustre: Skipped 2 previous similar messages [ 5450.742543] Lustre: Skipped 1 previous similar message [ 5453.417480] Lustre: server umount lustre-OST0000 complete [ 5455.841222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.843329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5455.848478] Lustre: Skipped 1 previous similar message [ 5455.856385] Lustre: Skipped 1 previous similar message [ 5460.963387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5460.970550] Lustre: Skipped 1 previous similar message [ 5461.782205] Lustre: server umount lustre-MDT0000 complete [ 5464.282635] LustreError: 150963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343310 with bad export cookie 3713470657117938104 [ 5464.288188] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.293032] LustreError: 150963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5464.455945] Lustre: server umount lustre-MDT0001 complete [ 5466.704390] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:48:31 (1780343311) [ 5467.645770] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5468.773034] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:48:33 (1780343313) [ 5473.056944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5473.307860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5475.400804] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5478.368701] LustreError: 154974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5478.376263] LustreError: 154974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5480.240246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5483.602928] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5486.846341] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5488.491656] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5492.568585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5496.007705] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5498.946815] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5514.867514] Lustre: server umount lustre-MDT0000 complete [ 5514.905303] LustreError: 154970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5515.232949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5515.243880] Lustre: Skipped 1 previous similar message [ 5516.773967] LustreError: 155696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343363 with bad export cookie 3713470657117938972 [ 5516.774745] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5516.779831] LustreError: 155696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5520.010806] LustreError: 155692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.028081] LustreError: 155692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5520.031386] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 5520.352863] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5520.362719] Lustre: Skipped 1 previous similar message [ 5525.129684] LustreError: 154970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.136455] LustreError: 154970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5530.252102] LustreError: 154969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5530.258904] LustreError: 154969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5531.103322] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5531.298254] Lustre: server umount lustre-MDT0001 complete [ 5535.248040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5543.246592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.387219] LustreError: 158496:0:(llog.c:1646:llog_backup()) MGC192.168.201.105@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5543.390320] Lustre: 158496:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.201.105@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5556.872157] LustreError: 158501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5577.351580] LustreError: 158503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5577.357641] LustreError: 158503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 5604.319322] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5604.322142] Lustre: 158496:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5612.168344] LustreError: 158520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5612.176318] LustreError: 158520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 5645.298330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5645.306103] Lustre: Skipped 2 previous similar messages [ 5647.194488] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5650.405968] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5651.418047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5661.663978] LustreError: 142814:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d96d9bab x1866825081137664/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5663.788499] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5666.792947] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5667.006628] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5668.368562] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5670.757221] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5671.855596] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5682.146174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5682.151051] LustreError: Skipped 1 previous similar message [ 5682.151178] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5682.153522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5682.153529] Lustre: Skipped 6 previous similar messages [ 5682.166092] Lustre: Skipped 1 previous similar message [ 5683.288818] Lustre: server umount lustre-MDT0000 complete [ 5685.091230] LustreError: 154954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343531 with bad export cookie 3713470657117945461 [ 5685.093623] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5685.095370] LustreError: 154954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5685.238553] Lustre: server umount lustre-MDT0001 complete [ 5696.809646] Lustre: server umount lustre-OST0000 complete [ 5707.266804] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5711.923503] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5730.123896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5730.280609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5730.296547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5730.340802] Lustre: lustre-MDT0000: new disk, initializing [ 5730.405113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5730.407336] Lustre: Skipped 1 previous similar message [ 5730.416162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5732.533914] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5738.926278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5738.986551] Lustre: 163316:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5739.112342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5740.909570] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5745.253451] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5747.261799] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5750.717864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5750.866755] Lustre: 165058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5753.848714] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5757.963784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5758.037438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5758.059175] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5760.019248] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5760.119820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5762.023146] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5762.131786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5763.043267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.046880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5763.047944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5763.047950] Lustre: Skipped 2 previous similar messages [ 5763.049209] Lustre: Skipped 2 previous similar messages [ 5763.059238] LustreError: Skipped 1 previous similar message [ 5765.109895] Lustre: server umount lustre-OST0000 complete [ 5773.052228] Lustre: server umount lustre-MDT0000 complete [ 5773.282631] LustreError: 163327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.292596] LustreError: 163327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 5775.025278] LustreError: 165057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343621 with bad export cookie 3713470657117962261 [ 5775.031016] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5775.038732] LustreError: 165057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5775.202158] Lustre: server umount lustre-MDT0001 complete [ 5779.511689] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:53:44 (1780343624) [ 5782.917391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5785.101208] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5788.881842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.758432] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5793.293659] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5795.063449] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5798.080038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5800.940480] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5804.951571] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5815.951204] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:54:21 (1780343661) [ 5818.851138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5818.859241] Lustre: Skipped 3 previous similar messages [ 5818.862546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5823.979797] Lustre: server umount lustre-OST0000 complete [ 5829.089548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.091924] Lustre: Skipped 5 previous similar messages [ 5832.422694] Lustre: server umount lustre-MDT0000 complete [ 5834.241153] LustreError: 168933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343680 with bad export cookie 3713470657117963150 [ 5834.243623] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5834.246310] LustreError: 168933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5834.437396] Lustre: server umount lustre-MDT0001 complete [ 5840.252315] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5841.849287] Key type lgssc unregistered [ 5842.007253] LNet: 170683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5842.013740] LNetError: 170683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5842.025528] LNet: Removed LNI 192.168.201.105@tcp [ 5842.444764] Key type .llcrypt unregistered [ 5842.446853] Key type ._llcrypt unregistered [ 5852.265350] Key type ._llcrypt registered [ 5852.267251] Key type .llcrypt registered [ 5852.340528] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5852.715937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5852.729835] alg: No test for adler32 (adler32-zlib) [ 5853.640235] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 5853.759258] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5855.343157] Key type lgssc registered [ 5855.933674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5859.332292] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5859.339410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5874.783715] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5879.903130] LustreError: 171888:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 5885.023341] LustreError: 171888:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5885.033289] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5885.035924] LustreError: MGC192.168.201.105@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5885.041806] LustreError: 171888:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5885.045701] LustreError: 171888:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 5885.048768] LustreError: 171888:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5885.091564] Lustre: server umount lustre-OST0000 complete [ 5885.093849] LustreError: 171888:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5889.991696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5891.226346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5893.186553] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5897.985687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5898.182525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5900.475359] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5903.006731] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5904.644973] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5908.211264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5908.301356] LustreError: 173971:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5908.308406] Lustre: 173971:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.201.105@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5908.375702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5910.675245] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5912.934410] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5913.775410] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.783055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5913.787422] Lustre: Skipped 1 previous similar message [ 5918.689796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5918.692936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5918.697966] Lustre: Skipped 1 previous similar message [ 5918.700505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5919.972536] Lustre: server umount lustre-MDT0000 complete [ 5921.801418] LustreError: 172366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343768 with bad export cookie 8178823484411006361 [ 5921.804687] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5921.808914] LustreError: 172366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5921.954129] Lustre: server umount lustre-MDT0001 complete [ 5934.255428] Lustre: server umount lustre-OST0000 complete [ 5938.457923] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5943.803791] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5961.648460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5961.803953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5961.826184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5961.880833] Lustre: lustre-MDT0000: new disk, initializing [ 5961.927395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5961.940707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5964.160576] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5969.794231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5969.847695] Lustre: 177344:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5969.871583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5969.873643] Lustre: Skipped 1 previous similar message [ 5969.910808] Lustre: lustre-MDT0001: new disk, initializing [ 5969.954043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5969.975695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5969.981715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5971.808781] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5975.631659] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5977.225875] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5980.148781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5980.264544] Lustre: lustre-OST0000: new disk, initializing [ 5980.266578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5980.270903] Lustre: 179088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5980.308132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5983.010509] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5986.964631] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5987.353580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5987.360983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5987.413982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5988.942788] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5989.038889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5990.776503] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5990.876559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5992.423705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5992.431902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.446279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5992.449937] Lustre: Skipped 4 previous similar messages [ 5994.071623] Lustre: server umount lustre-OST0000 complete [ 5997.536056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5997.539655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5997.542439] Lustre: Skipped 2 previous similar messages [ 6002.657098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6002.660589] Lustre: Skipped 2 previous similar messages [ 6010.335643] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6010.414301] Lustre: server umount lustre-MDT0000 complete [ 6011.874480] LustreError: 178586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6011.886455] LustreError: 178586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6012.321996] LustreError: 178124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343858 with bad export cookie 8178823484411007040 [ 6012.329376] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6012.330799] LustreError: 178124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6012.517629] Lustre: server umount lustre-MDT0001 complete [ 6016.719130] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:57:42 (1780343862) [ 6024.941641] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6026.449403] Key type lgssc unregistered [ 6026.638215] LNet: 181864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6026.644580] LNetError: 181864:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6026.660221] LNet: Removed LNI 192.168.201.105@tcp [ 6027.078561] Key type .llcrypt unregistered [ 6027.080690] Key type ._llcrypt unregistered [ 6036.140929] Key type ._llcrypt registered [ 6036.143832] Key type .llcrypt registered [ 6036.218381] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6036.921959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6036.934223] alg: No test for adler32 (adler32-zlib) [ 6037.828413] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 6037.961313] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6039.631149] Key type lgssc registered [ 6040.247312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6044.036161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6044.044278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6045.336557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6047.233200] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6050.977959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.126632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6052.679924] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6054.935847] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6056.629263] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6060.178311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6060.273300] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6060.275202] LustreError: 184678:0:(llog.c:1687:llog_backup()) MGC192.168.201.105@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6060.279899] Lustre: 184678:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.201.105@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6060.332502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.750145] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6065.137065] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6066.657753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6066.658146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6066.659321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6066.670298] Lustre: Skipped 2 previous similar messages [ 6071.780319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6071.784193] Lustre: Skipped 4 previous similar messages [ 6072.317190] Lustre: server umount lustre-MDT0000 complete [ 6073.841483] LustreError: 183759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343920 with bad export cookie 10692878990752546157 [ 6073.841900] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6073.847557] LustreError: 183759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6073.983841] Lustre: server umount lustre-MDT0001 complete [ 6085.924923] Lustre: server umount lustre-OST0000 complete [ 6089.633205] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6093.868423] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6109.651814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6109.746413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6109.763966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6109.803566] Lustre: lustre-MDT0000: new disk, initializing [ 6109.841762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6109.851666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6111.421294] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6116.286508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.330413] Lustre: 188051:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6116.351066] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6116.352362] Lustre: Skipped 1 previous similar message [ 6116.353914] LustreError: 188817:0:(llog.c:1687:llog_backup()) MGC192.168.201.105@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6116.357356] LustreError: 188817:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 6116.359633] Lustre: 188817:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.201.105@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6116.368228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6116.370406] Lustre: Skipped 1 previous similar message [ 6116.405928] Lustre: lustre-MDT0001: new disk, initializing [ 6116.443716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6116.457216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6116.463280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6117.914951] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6121.474897] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6122.976592] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6125.682668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6125.785178] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6125.786749] Lustre: Skipped 1 previous similar message [ 6125.788438] LustreError: 189770:0:(llog.c:1687:llog_backup()) MGC192.168.201.105@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6125.791688] LustreError: 189770:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 6125.794064] Lustre: 189770:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.201.105@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6125.797572] Lustre: 189770:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6125.807280] Lustre: lustre-OST0000: new disk, initializing [ 6125.809741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6125.813532] Lustre: 189794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6125.843237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6128.084054] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6130.166469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6130.170024] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6130.207954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6131.389492] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6133.022663] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6133.119412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6134.971634] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6135.065683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6140.385688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.386154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6140.391726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6140.396454] Lustre: Skipped 1 previous similar message [ 6142.038499] Lustre: server umount lustre-OST0000 complete [ 6145.505737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6145.507048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6145.511172] Lustre: Skipped 2 previous similar messages [ 6145.515054] Lustre: Skipped 1 previous similar message [ 6150.625957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6150.628701] Lustre: Skipped 1 previous similar message [ 6158.303117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6158.361516] Lustre: server umount lustre-MDT0000 complete [ 6159.841301] LustreError: 189260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6159.848373] LustreError: 189260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6159.952174] LustreError: 188830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344006 with bad export cookie 10692878990752546836 [ 6159.958042] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6159.958455] LustreError: 188830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6160.104951] Lustre: server umount lustre-MDT0001 complete [ 6163.944126] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:00:09 (1780344009) [ 6170.295177] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6174.430993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.614322] LustreError: 192752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6174.629884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6176.181890] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6179.413864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6181.011325] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6182.141309] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6184.722952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6186.962364] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6190.082539] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6190.142911] Lustre: lustre-OST0001: new disk, initializing [ 6190.144850] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6190.147416] Lustre: 195004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6190.800141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6190.804787] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6190.852829] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6192.551894] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6198.901853] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6201.975829] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6202.802936] Lustre: DEBUG MARKER: ADD OST3 [ 6204.797124] loop: module loaded [ 6204.819411] loop0: detected capacity change from 0 to 204800000 [ 6204.911640] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6204.916937] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6204.924214] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6207.303749] loop0: detected capacity change from 0 to 204800000 [ 6209.613195] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6209.687689] Lustre: lustre-OST0003: new disk, initializing [ 6209.690189] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6209.693655] Lustre: 196718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6209.738267] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6209.742773] Lustre: Skipped 3 previous similar messages [ 6210.052956] Lustre: DEBUG MARKER: STOP OST3 [ 6210.861568] Lustre: Failing over lustre-OST0003 [ 6210.911974] Lustre: server umount lustre-OST0003 complete [ 6211.023931] LustreError: 195107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6212.483515] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6216.164786] LustreError: 194198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6216.174695] LustreError: 194198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6216.846795] Lustre: DEBUG MARKER: START OST3 again [ 6219.970057] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6222.757841] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6225.396588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6225.402461] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6225.420596] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 6237.553043] Lustre: Failing over lustre-OST0003 [ 6237.621784] Lustre: server umount lustre-OST0003 complete [ 6240.742596] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6240.744424] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6240.746653] LustreError: Skipped 2 previous similar messages [ 6240.752919] LustreError: 194201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6240.761937] LustreError: 194201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6250.979453] LustreError: 194198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6250.979558] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6250.981440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6250.981448] Lustre: Skipped 2 previous similar messages [ 6250.990331] Lustre: Skipped 3 previous similar messages [ 6250.995238] LustreError: 194198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6265.311510] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6265.400632] Lustre: server umount lustre-MDT0000 complete [ 6267.206139] LustreError: 192732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344113 with bad export cookie 10692878990752547725 [ 6267.209996] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.211694] LustreError: 192732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6271.456673] LustreError: 192749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6271.458040] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.459474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6271.459479] Lustre: Skipped 11 previous similar messages [ 6271.463358] LustreError: 192749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 6271.468876] Lustre: Skipped 2 previous similar messages [ 6273.495224] Lustre: server umount lustre-MDT0001 complete [ 6275.062552] Lustre: server umount lustre-OST0000 complete [ 6276.683126] Lustre: server umount lustre-OST0001 complete [ 6279.403920] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6283.087180] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6299.816447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6299.921790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6299.937252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6299.979242] Lustre: lustre-MDT0000: new disk, initializing [ 6300.021714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6300.026399] Lustre: Skipped 1 previous similar message [ 6300.037187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6301.945456] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6307.554936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.612775] Lustre: 202070:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6307.761431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6309.541053] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6313.467670] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6314.968632] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6317.826190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6317.944591] Lustre: lustre-OST0000: new disk, initializing [ 6317.947611] Lustre: Skipped 1 previous similar message [ 6317.949967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6317.952663] Lustre: Skipped 2 previous similar messages [ 6317.955338] Lustre: 203813:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6319.489557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6319.494295] Lustre: Skipped 1 previous similar message [ 6319.496806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6319.543124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6320.489220] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6324.363882] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6326.340943] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6326.430548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6328.046287] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6328.125768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6329.827770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6329.828992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.834542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6329.838223] Lustre: Skipped 1 previous similar message [ 6329.844767] Lustre: Skipped 4 previous similar messages [ 6343.136182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6343.195981] Lustre: server umount lustre-OST0000 complete [ 6350.818204] Lustre: server umount lustre-MDT0000 complete [ 6352.300324] LustreError: 202062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344198 with bad export cookie 10692878990752549958 [ 6352.304599] LustreError: 202062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6352.304627] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6352.453968] Lustre: server umount lustre-MDT0001 complete [ 6354.008071] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:03:19 (1780344199) [ 6360.963858] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6365.046514] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6378.159491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.263347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6378.277630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6378.314338] Lustre: lustre-MDT0000: new disk, initializing [ 6378.359387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6380.008397] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6384.640223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.684916] Lustre: 208232:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6384.800702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6386.251183] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6389.792956] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6391.201725] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6393.996989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6394.108365] Lustre: 209974:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6395.846331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6396.647958] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6400.395476] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6402.040837] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6402.128643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6403.751654] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6403.851552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6406.116398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6406.116489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.119491] LustreError: Skipped 2 previous similar messages [ 6406.124883] Lustre: Skipped 2 previous similar messages [ 6406.127481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6406.130506] Lustre: Skipped 9 previous similar messages [ 6410.848232] Lustre: server umount lustre-OST0000 complete [ 6415.843730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6418.702789] Lustre: server umount lustre-MDT0000 complete [ 6420.281332] LustreError: 208224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344266 with bad export cookie 10692878990752550840 [ 6420.283854] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6420.285586] LustreError: 208224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6420.419991] Lustre: server umount lustre-MDT0001 complete [ 6423.678432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6423.854709] LustreError: 212041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6423.862588] LustreError: 212041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6425.383748] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6428.596399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6428.794288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6428.798736] Lustre: Skipped 6 previous similar messages [ 6430.506297] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6432.558702] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6433.985151] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6436.576804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6438.936255] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6441.028736] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6459.361445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6459.363985] LustreError: Skipped 1 previous similar message [ 6464.082897] Lustre: server umount lustre-OST0000 complete [ 6471.578685] Lustre: server umount lustre-MDT0000 complete [ 6472.902177] LustreError: 212024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344319 with bad export cookie 10692878990752551722 [ 6472.905549] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6472.906330] LustreError: 212024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6473.039120] Lustre: server umount lustre-MDT0001 complete [ 6576.421448] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6577.464646] Key type lgssc unregistered [ 6577.605354] LNet: 215364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6577.611410] LNetError: 215364:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6577.623444] LNet: Removed LNI 192.168.201.105@tcp [ 6577.946118] Key type .llcrypt unregistered [ 6577.947038] Key type ._llcrypt unregistered [ 6586.946887] Key type ._llcrypt registered [ 6586.948312] Key type .llcrypt registered [ 6586.991118] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6593.169403] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6593.479328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6593.525077] alg: No test for adler32 (adler32-zlib) [ 6594.385897] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 6594.479715] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6596.063160] Key type lgssc registered [ 6596.463754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6609.502382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6609.510664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6610.644943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6610.656852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6610.695993] Lustre: lustre-MDT0000: new disk, initializing [ 6610.724084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6610.731789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6612.186962] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6616.956253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.006057] Lustre: 218653:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6617.035809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6617.038553] Lustre: Skipped 1 previous similar message [ 6617.085949] Lustre: lustre-MDT0001: new disk, initializing [ 6617.139753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6617.159687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6617.164101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6618.754940] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6622.244677] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6623.582163] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6626.027702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6626.138991] Lustre: lustre-OST0000: new disk, initializing [ 6626.141349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6626.144681] Lustre: 220396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6626.174987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6628.345272] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6630.918657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6630.923232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6630.964629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6631.777140] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6633.325702] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6633.421698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6634.939273] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6635.042743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6636.001017] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.001076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6636.003251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6636.006284] Lustre: Skipped 1 previous similar message [ 6637.838978] Lustre: server umount lustre-OST0000 complete [ 6641.120593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6641.121234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6641.126090] Lustre: Skipped 1 previous similar message [ 6641.129414] Lustre: Skipped 2 previous similar messages [ 6645.290963] Lustre: server umount lustre-MDT0000 complete [ 6646.658215] LustreError: 219434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344493 with bad export cookie 9865894251070872267 [ 6646.661736] LustreError: 219434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6646.661835] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.787968] Lustre: server umount lustre-MDT0001 complete [ 6653.000727] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6656.468631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.600791] LustreError: 223158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6656.608036] LustreError: 223158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6656.617876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6657.920943] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6661.037286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6661.160950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6662.485019] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6663.477723] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6665.798000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6668.078810] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6671.373930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6671.448862] Lustre: lustre-OST0001: new disk, initializing [ 6671.451315] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6671.458960] Lustre: 225411:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6671.499414] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6671.502733] Lustre: Skipped 1 previous similar message [ 6673.719373] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6678.527161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6678.530456] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6678.566438] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6679.200948] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6684.840629] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6688.823780] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:08:54 (1780344534) [ 6691.808152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6691.811015] LustreError: Skipped 1 previous similar message [ 6691.812327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.816505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6693.856398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6693.858149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6693.865084] Lustre: Skipped 1 previous similar message [ 6695.642224] Lustre: server umount lustre-MDT0000 complete [ 6696.953167] LustreError: 223139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344543 with bad export cookie 9865894251070873149 [ 6696.956536] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6696.957954] LustreError: 223139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6697.068509] Lustre: server umount lustre-MDT0001 complete [ 6708.167547] Lustre: server umount lustre-OST0000 complete [ 6718.747109] Lustre: server umount lustre-OST0001 complete [ 6721.095873] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6724.845703] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6739.146640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.251370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6739.274439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6739.318604] Lustre: lustre-MDT0000: new disk, initializing [ 6739.347865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6739.359902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6740.896326] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6745.103667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6745.158144] Lustre: 229304:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6745.276899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6746.745297] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6750.068839] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6751.371661] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6753.870352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6753.970458] Lustre: lustre-OST0000: new disk, initializing [ 6753.972580] Lustre: Skipped 1 previous similar message [ 6753.974947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6753.977220] Lustre: Skipped 2 previous similar messages [ 6753.979033] Lustre: 231045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6755.513465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6755.516078] Lustre: Skipped 1 previous similar message [ 6755.519128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6755.551415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6755.961541] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6759.357256] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6760.889345] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6760.977387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6762.428788] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6762.508754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6766.052812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6766.053107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6766.057224] Lustre: Skipped 1 previous similar message [ 6766.062250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.064655] Lustre: Skipped 4 previous similar messages [ 6769.303667] Lustre: server umount lustre-OST0000 complete [ 6771.169380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6771.178419] LustreError: Skipped 1 previous similar message [ 6776.288075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6776.290671] Lustre: Skipped 3 previous similar messages [ 6784.991151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6785.056213] Lustre: server umount lustre-MDT0000 complete [ 6786.324214] LustreError: 229295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344632 with bad export cookie 9865894251070875270 [ 6786.327828] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6786.330556] LustreError: 229295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6786.444815] Lustre: server umount lustre-MDT0001 complete [ 6789.551255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6789.725116] LustreError: 233113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6789.753389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6789.756404] Lustre: Skipped 2 previous similar messages [ 6791.199215] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6794.202965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6795.816818] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6797.819466] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6799.185100] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6801.570530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6803.706121] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6805.794861] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6952.416059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6952.418903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6952.423259] Lustre: Skipped 3 previous similar messages [ 6952.425640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6952.428038] Lustre: Skipped 2 previous similar messages [ 6965.215130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6965.271413] Lustre: server umount lustre-OST0000 complete [ 6968.287913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6968.291120] LustreError: Skipped 1 previous similar message [ 6969.825095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.828954] Lustre: Skipped 2 previous similar messages [ 6972.639744] Lustre: server umount lustre-MDT0000 complete [ 6974.123144] LustreError: 233095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344820 with bad export cookie 9865894251070876152 [ 6974.126234] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6974.127640] LustreError: 233095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6974.245166] Lustre: server umount lustre-MDT0001 complete [ 7080.559141] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7081.584061] Key type lgssc unregistered [ 7081.706364] LNet: 236362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7081.709420] LNetError: 236362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7081.717298] LNet: Removed LNI 192.168.201.105@tcp [ 7082.013130] Key type .llcrypt unregistered [ 7082.014694] Key type ._llcrypt unregistered [ 7089.793280] Key type ._llcrypt registered [ 7089.794305] Key type .llcrypt registered [ 7089.827235] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7095.729110] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7096.019453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7096.030262] alg: No test for adler32 (adler32-zlib) [ 7096.875255] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 7096.956054] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7098.535134] Key type lgssc registered [ 7098.860071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7112.920247] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7116.143632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7116.148049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.222655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7117.232117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7117.262927] Lustre: lustre-MDT0000: new disk, initializing [ 7117.283152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7117.289915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7118.454126] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7122.376414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.412692] Lustre: 240352:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7122.431925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7122.433485] Lustre: Skipped 1 previous similar message [ 7122.464219] Lustre: lustre-MDT0001: new disk, initializing [ 7122.493737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7122.504826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7122.507337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7123.699557] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7125.863365] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7128.005548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7128.121971] Lustre: lustre-OST0000: new disk, initializing [ 7128.124425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7128.128826] Lustre: 241950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7128.168321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7130.160388] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7134.185028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7134.237579] Lustre: lustre-OST0001: new disk, initializing [ 7134.239319] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7134.241430] Lustre: 242795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7134.271908] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7135.808495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7135.812906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7135.856664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7135.864383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7136.126523] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7141.455354] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7148.084210] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7151.585679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7151.586594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7151.589650] Lustre: Skipped 2 previous similar messages [ 7151.593336] Lustre: Skipped 3 previous similar messages [ 7156.704173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7156.707123] Lustre: Skipped 3 previous similar messages [ 7156.906183] Lustre: server umount lustre-MDT0000 complete [ 7158.144199] LustreError: 243612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345004 with bad export cookie 7434375269235557217 [ 7158.147655] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7158.147912] LustreError: 243612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7158.257594] Lustre: server umount lustre-MDT0001 complete [ 7169.030134] Lustre: server umount lustre-OST0000 complete [ 7180.505164] Lustre: server umount lustre-OST0001 complete [ 7184.976279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7187.909182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7190.559884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7193.592682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7199.554309] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7203.010805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7203.017772] Lustre: lustre-MDT0000: reset Object Index mappings [ 7203.178651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7204.429110] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7207.331650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.340303] Lustre: lustre-MDT0001: reset Object Index mappings [ 7208.807912] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7209.812324] Lustre: 248877:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7212.082026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7212.212754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7212.215180] Lustre: Skipped 1 previous similar message [ 7213.221267] LustreError: 249231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7213.226425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7213.231895] LustreError: 249231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7214.253925] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7217.128036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7218.283639] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7219.111949] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7222.186493] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7223.336096] Lustre: 250710:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7227.871863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7227.874535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.878350] Lustre: Skipped 1 previous similar message [ 7227.880207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7228.900212] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7228.903742] Lustre: Skipped 1 previous similar message [ 7230.613188] Lustre: server umount lustre-MDT0000 complete [ 7231.802172] LustreError: 247748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345078 with bad export cookie 7434375269235563895 [ 7231.805089] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7231.806393] LustreError: 247748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7231.915412] Lustre: server umount lustre-MDT0001 complete [ 7243.200996] Lustre: server umount lustre-OST0000 complete [ 7253.722444] Lustre: server umount lustre-OST0001 complete [ 7259.889582] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:18:25 (1780345105) [ 7265.263801] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7268.419186] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7280.210190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7280.297762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7280.308533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7280.346043] Lustre: lustre-MDT0000: new disk, initializing [ 7280.372424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7280.374662] Lustre: Skipped 1 previous similar message [ 7280.380796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7280.384540] Lustre: Skipped 1 previous similar message [ 7281.633062] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7285.706903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7285.746979] Lustre: 254920:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7285.850685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7285.855056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7285.858315] Lustre: Skipped 1 previous similar message [ 7287.096961] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7290.145118] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7291.368041] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7293.594977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7293.694602] Lustre: lustre-OST0000: new disk, initializing [ 7293.696009] Lustre: Skipped 1 previous similar message [ 7293.697571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7293.699763] Lustre: Skipped 2 previous similar messages [ 7293.701968] Lustre: 256662:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7295.287034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7295.291250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7295.328569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7295.596700] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7298.589080] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7299.889679] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7299.957564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7301.248907] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7301.323078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7305.698150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7305.698459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7305.699187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7305.699191] Lustre: Skipped 4 previous similar messages [ 7305.701156] LustreError: Skipped 1 previous similar message [ 7305.704532] Lustre: Skipped 2 previous similar messages [ 7308.056583] Lustre: server umount lustre-OST0000 complete [ 7310.816548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.817186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7310.820680] Lustre: Skipped 1 previous similar message [ 7310.822781] Lustre: Skipped 1 previous similar message [ 7315.410918] Lustre: server umount lustre-MDT0000 complete [ 7315.935964] LustreError: 254932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.941117] LustreError: 254932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7316.589071] LustreError: 256661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345162 with bad export cookie 7434375269235565967 [ 7316.592090] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7316.593776] LustreError: 256661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7316.709603] Lustre: server umount lustre-MDT0001 complete [ 7322.452584] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7325.869464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7326.030095] LustreError: 259425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.045743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7326.048374] Lustre: Skipped 2 previous similar messages [ 7327.442117] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7330.274194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.695908] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7332.655421] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7334.855436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7336.924625] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7339.712779] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7339.765739] Lustre: lustre-OST0001: new disk, initializing [ 7339.767750] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7339.770864] Lustre: 261678:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7340.988099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7340.991497] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7341.027791] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7341.572838] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7345.844590] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7346.922113] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7348.594959] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7348.600469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7350.751737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7350.754156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7350.757840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7350.760409] Lustre: Skipped 1 previous similar message [ 7356.311667] Lustre: server umount lustre-MDT0000 complete [ 7356.385883] LustreError: 259420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7356.390567] LustreError: 259420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7357.548284] LustreError: 259408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345203 with bad export cookie 7434375269235566849 [ 7357.550953] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7357.551602] LustreError: 259408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7357.664859] Lustre: server umount lustre-MDT0001 complete [ 7368.709084] Lustre: server umount lustre-OST0000 complete [ 7380.188699] Lustre: server umount lustre-OST0001 complete [ 7385.199741] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7388.577018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7388.706329] LustreError: 264694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.076282] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7392.807525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7392.935954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7392.937918] Lustre: Skipped 4 previous similar messages [ 7394.218012] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7395.112136] Lustre: 265800:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7397.203201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7399.097197] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7399.394267] LustreError: 266154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7399.401423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7401.809695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7402.925750] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7403.721465] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7406.742466] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7407.871021] Lustre: 267642:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7409.538243] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7409.540098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7412.470264] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:20:58 (1780345258) [ 7423.967809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7423.972488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7423.975839] Lustre: Skipped 3 previous similar messages [ 7423.977483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7423.979053] Lustre: Skipped 4 previous similar messages [ 7430.035302] Lustre: server umount lustre-MDT0000 complete [ 7431.212780] LustreError: 267645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345277 with bad export cookie 7434375269235573142 [ 7431.215264] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7431.216010] LustreError: 267645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7433.696099] LustreError: 267132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7433.701029] LustreError: 267132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7437.357508] Lustre: server umount lustre-MDT0001 complete [ 7444.674319] Lustre: server umount lustre-OST0000 complete [ 7460.319096] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7460.384042] Lustre: server umount lustre-OST0001 complete [ 7462.337106] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7465.430952] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7480.694143] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7484.151477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7484.237415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7484.249107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7484.288307] Lustre: lustre-MDT0000: new disk, initializing [ 7484.319668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7485.642565] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7489.816183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7489.857629] Lustre: 271824:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7489.963975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7491.389216] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7495.984496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7496.088543] Lustre: 273425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7496.573289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7498.122034] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7502.358232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7504.156832] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7504.317283] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7508.596843] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7513.639121] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:22:39 (1780345359) [ 7514.592684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.594963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7514.597165] Lustre: Skipped 8 previous similar messages [ 7514.602915] Lustre: Skipped 11 previous similar messages [ 7520.477892] Lustre: server umount lustre-MDT0000 complete [ 7521.710143] LustreError: 272604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345368 with bad export cookie 7434375269235581010 [ 7521.713411] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7521.714582] LustreError: 272604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7521.829766] Lustre: server umount lustre-MDT0001 complete [ 7533.058045] Lustre: server umount lustre-OST0000 complete [ 7544.480190] Lustre: server umount lustre-OST0001 complete [ 7547.451863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7547.625212] LustreError: 276503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.631048] LustreError: 276503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7547.654678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7547.656695] Lustre: Skipped 6 previous similar messages [ 7548.970631] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7551.795662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7553.195420] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7554.938389] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7556.180463] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7558.590464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7560.498316] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7562.172601] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7583.811477] loop1: detected capacity change from 0 to 204800000 [ 7583.888456] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7583.891997] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7583.895685] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7585.502215] loop1: detected capacity change from 0 to 204800000 [ 7587.425254] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7587.456659] LustreError: 276492:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs11937: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7587.465043] LustreError: 276492:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs11937 [ 7587.467589] LustreError: 276492:0:(mgs_handler.c:517:mgs_target_reg()) fs11937-MDT0001: Error checking target index: rc = -13 [ 7587.471156] LustreError: fs11937-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7587.474463] LustreError: 279948:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 7587.477022] LustreError: 279948:0:(tgt_mount.c:2084:server_put_super()) no obd fs11937-MDT0001 [ 7587.479300] LustreError: 279948:0:(tgt_mount.c:116:server_deregister_mount()) fs11937-MDT0001 not registered [ 7587.490510] Lustre: server umount fs11937-MDT0001 complete [ 7587.492037] LustreError: 279948:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7590.254652] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7590.285994] Lustre: Setting parameter fs11937-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs11937-MDT0001 [ 7590.288932] Lustre: Skipped 2 previous similar messages [ 7590.304263] Lustre: srv-fs11937-MDT0001: No data found on store. Initialize space. [ 7590.306017] Lustre: Skipped 4 previous similar messages [ 7590.340405] Lustre: fs11937-MDT0001: new disk, initializing [ 7590.341752] Lustre: Skipped 3 previous similar messages [ 7590.371645] Lustre: srv-fs11937-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7590.374235] Lustre: 280414:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11937-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7590.377200] LustreError: 280414:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11937-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7591.655767] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7594.525648] Lustre: server umount fs11937-MDT0001 complete [ 7598.781960] loop1: detected capacity change from 0 to 204800000 [ 7600.627785] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7600.730729] Lustre: 281963:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11937-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7600.733341] LustreError: 281963:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11937-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7601.955272] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7603.590080] Lustre: server umount fs11937-MDT0001 complete [ 7608.287865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7608.290598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7608.295170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7608.297225] Lustre: Skipped 4 previous similar messages [ 7613.575556] LustreError: 278105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7613.584299] LustreError: 278105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 7621.087112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7621.144809] Lustre: server umount lustre-MDT0000 complete [ 7623.703415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7623.736741] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7624.996389] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7627.309620] loop1: detected capacity change from 0 to 204800000 [ 7629.017237] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.114972] Lustre: 284371:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11937-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7629.117270] LustreError: 284371:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11937-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7629.285844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7630.248678] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7631.746765] Lustre: server umount fs11937-MDT0001 complete [ 7638.495887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7644.238513] Lustre: server umount lustre-OST0000 complete [ 7651.548509] Lustre: server umount lustre-MDT0000 complete [ 7652.736306] LustreError: 276485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345499 with bad export cookie 7434375269235583684 [ 7652.740056] LustreError: 276485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7652.853163] Lustre: server umount lustre-MDT0001 complete [ 7656.131365] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7657.087681] Key type lgssc unregistered [ 7657.216545] LNet: 286675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7657.219194] LNetError: 286675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7657.227293] LNet: Removed LNI 192.168.201.105@tcp [ 7657.527102] Key type .llcrypt unregistered [ 7657.528276] Key type ._llcrypt unregistered [ 7664.893831] Key type ._llcrypt registered [ 7664.894755] Key type .llcrypt registered [ 7664.935527] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7665.267091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7665.316562] alg: No test for adler32 (adler32-zlib) [ 7666.163140] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 7666.233855] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7667.815194] Key type lgssc registered [ 7668.180420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7670.708653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7670.715561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7671.862496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7673.110466] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7675.799990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.920144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7677.119603] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7678.769399] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7679.865209] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7681.952300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7682.016872] LustreError: 289506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7682.022123] LustreError: 289506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7682.062307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7683.774534] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7685.428858] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7686.178264] LustreError: 289509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.183362] LustreError: 289509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7691.236046] LustreError: 290065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7691.241727] LustreError: 290065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7696.354325] LustreError: 289509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.358681] LustreError: 289509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7697.637927] loop1: detected capacity change from 0 to 204800000 [ 7697.692560] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7697.695806] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7697.699287] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7699.317815] loop1: detected capacity change from 0 to 204800000 [ 7701.056270] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7701.101185] Lustre: fs30080-OST0000: new disk, initializing [ 7701.103075] Lustre: srv-fs30080-OST0000: No data found on store. Initialize space. [ 7701.105803] Lustre: 291347:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7701.128708] Lustre: fs30080-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.477034] LustreError: 290064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7701.484240] LustreError: 290064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7702.943560] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7706.314319] Lustre: server umount fs30080-OST0000 complete [ 7710.223584] loop1: detected capacity change from 0 to 204800000 [ 7711.715132] LustreError: 289506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.720699] LustreError: 289506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7712.076380] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7712.138172] Lustre: fs30080-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7714.098492] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7716.770983] loop2: detected capacity change from 0 to 204800000 [ 7716.843574] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7716.846928] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7716.850040] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7718.512105] loop2: detected capacity change from 0 to 204800000 [ 7720.275799] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7720.302430] LustreError: fs30080-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7720.305984] LustreError: fs30080-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7720.308229] LustreError: 294350:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 7720.310082] LustreError: 294350:0:(tgt_mount.c:2084:server_put_super()) no obd fs30080-OST0001 [ 7720.311570] LustreError: 294350:0:(tgt_mount.c:116:server_deregister_mount()) fs30080-OST0001 not registered [ 7720.321426] Lustre: server umount fs30080-OST0001 complete [ 7720.322960] LustreError: 294350:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7723.792281] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7723.836413] Lustre: fs30080-OST0001: new disk, initializing [ 7723.838337] Lustre: srv-fs30080-OST0001: No data found on store. Initialize space. [ 7723.841156] Lustre: 294901:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7723.864640] Lustre: fs30080-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7725.663968] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7727.752086] LustreError: 290066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7727.758076] LustreError: 290066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7729.024292] Lustre: server umount fs30080-OST0000 complete [ 7731.997897] Lustre: server umount fs30080-OST0001 complete [ 7735.931535] loop1: detected capacity change from 0 to 204800000 [ 7737.707664] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7739.523876] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7741.720876] loop2: detected capacity change from 0 to 204800000 [ 7743.553152] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7743.603492] Lustre: fs30080-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7743.605652] Lustre: Skipped 1 previous similar message [ 7745.395700] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7748.149223] Lustre: server umount fs30080-OST0000 complete [ 7751.141383] Lustre: server umount fs30080-OST0001 complete [ 7757.793392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7757.794102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7757.797079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7757.800904] LustreError: Skipped 1 previous similar message [ 7757.802612] Lustre: Skipped 1 previous similar message [ 7762.913086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7762.915645] LustreError: 289508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7762.920253] LustreError: 289508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 7768.033592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7768.035806] Lustre: Skipped 2 previous similar messages [ 7769.055107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7769.110393] Lustre: server umount lustre-OST0000 complete [ 7773.152685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7773.152780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7773.156678] Lustre: Skipped 1 previous similar message [ 7773.157513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7773.161576] Lustre: Skipped 1 previous similar message [ 7776.477881] Lustre: server umount lustre-MDT0000 complete [ 7777.814496] LustreError: 287887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345624 with bad export cookie 2324161161020948085 [ 7777.817987] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7777.819387] LustreError: 287887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7777.926528] Lustre: server umount lustre-MDT0001 complete [ 7781.322285] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7782.289102] Key type lgssc unregistered [ 7782.421385] LNet: 300744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7782.424508] LNetError: 300744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7782.432321] LNet: Removed LNI 192.168.201.105@tcp [ 7782.711117] Key type .llcrypt unregistered [ 7782.712054] Key type ._llcrypt unregistered [ 7792.142681] Key type ._llcrypt registered [ 7792.143596] Key type .llcrypt registered [ 7792.175597] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7797.858554] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7798.297779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7798.304416] alg: No test for adler32 (adler32-zlib) [ 7799.148708] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 7799.231941] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7800.807176] Key type lgssc registered [ 7801.151275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7812.344738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7812.351172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7813.427417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7813.437351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7813.471853] Lustre: lustre-MDT0000: new disk, initializing [ 7813.494769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7813.500094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7814.760192] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7818.675784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.715955] Lustre: 304540:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7818.739811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7818.741821] Lustre: Skipped 1 previous similar message [ 7818.779236] Lustre: lustre-MDT0001: new disk, initializing [ 7818.810592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7818.821906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7818.825665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7820.018383] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7823.003819] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7824.213450] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7826.442540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7826.538674] Lustre: lustre-OST0000: new disk, initializing [ 7826.540345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7826.542894] Lustre: 306279:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7826.570687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.510926] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7831.688437] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7832.058234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7832.061169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7832.096153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7832.422718] Lustre: Mounted lustre-client [ 7832.737990] LustreError: 307029:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8ec0bb6ae800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 7832.765202] Lustre: Unmounted lustre-client [ 7837.151989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7837.152414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7837.154148] LustreError: Skipped 1 previous similar message [ 7837.157756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7837.160170] Lustre: Skipped 1 previous similar message [ 7841.803081] Lustre: server umount lustre-OST0000 complete [ 7844.320807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7844.323763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7844.327968] Lustre: Skipped 1 previous similar message [ 7844.330078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7847.904493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7847.904627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.176033] Lustre: server umount lustre-MDT0000 complete [ 7851.561173] LustreError: 304530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345697 with bad export cookie 2883023854710139224 [ 7851.568621] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7851.801663] Lustre: server umount lustre-MDT0001 complete [ 7859.731508] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7861.509821] Key type lgssc unregistered [ 7891.487602] LNet: 308306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7891.492200] LNetError: 308306:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7964.389348] LNet: Removed LNI 192.168.201.105@tcp [ 7964.767123] Key type .llcrypt unregistered [ 7964.769348] Key type ._llcrypt unregistered [ 7974.196428] Key type ._llcrypt registered [ 7974.198352] Key type .llcrypt registered [ 7974.253407] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7983.710164] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7984.348423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7984.367458] alg: No test for adler32 (adler32-zlib) [ 7985.273694] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 7985.397220] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7987.023176] Key type lgssc registered [ 7987.826615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8005.898774] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:30:51 (1780345851) [ 8009.760816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8009.769906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.893124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8010.920736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8010.978152] Lustre: lustre-MDT0000: new disk, initializing [ 8011.019226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8011.028727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8013.216577] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8019.361789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.423850] Lustre: 311472:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8019.459546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8019.463059] Lustre: Skipped 1 previous similar message [ 8019.506345] Lustre: lustre-MDT0001: new disk, initializing [ 8019.558746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8019.577041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8019.586165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8021.927315] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8026.386563] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8028.083147] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8031.850551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8032.001475] Lustre: lustre-OST0000: new disk, initializing [ 8032.003613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8032.006486] Lustre: 313212:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8032.057887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8034.666553] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8037.905367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8037.915712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8037.989209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8039.242757] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8048.109846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8048.110896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.112432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8048.113304] LustreError: Skipped 1 previous similar message [ 8048.132671] Lustre: Skipped 1 previous similar message [ 8051.528436] Lustre: server umount lustre-OST0000 complete [ 8055.265963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8055.270815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8055.283639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8055.286450] Lustre: Skipped 1 previous similar message [ 8058.337503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8058.338739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8059.880175] Lustre: server umount lustre-MDT0000 complete [ 8062.028345] LustreError: 311462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345908 with bad export cookie 3884450295387792293 [ 8062.034218] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8062.179997] Lustre: server umount lustre-MDT0001 complete [ 8067.312491] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8068.778336] Key type lgssc unregistered [ 8068.943759] LNet: 314841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8068.946662] LNetError: 314841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8069.998772] LNet: Removed LNI 192.168.201.105@tcp [ 8070.472839] Key type .llcrypt unregistered [ 8070.474353] Key type ._llcrypt unregistered [ 8084.579444] Key type ._llcrypt registered [ 8084.580486] Key type .llcrypt registered [ 8084.644635] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8085.202607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8085.214778] alg: No test for adler32 (adler32-zlib) [ 8086.093609] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 8086.177677] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8087.767128] Key type lgssc registered [ 8088.173439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8091.244332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8091.257115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8092.463728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8093.957336] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8097.068280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8097.203924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8098.672406] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8100.563592] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8101.872887] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8104.330069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8104.447165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8106.614001] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8108.734409] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8123.426222] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8124.091604] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8125.583353] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 8125.667501] Lustre: server umount lustre-MDT0000 complete [ 8126.946421] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8126.947058] LustreError: 316218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8126.953304] Lustre: Skipped 1 previous similar message [ 8126.965540] LustreError: 316218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8127.130299] LustreError: 316205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345973 with bad export cookie 15713718621353912130 [ 8127.130888] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8127.138152] LustreError: 316205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8130.697667] LustreError: 316219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8130.705337] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 8132.063931] LustreError: 316220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.064106] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8132.073487] Lustre: Skipped 1 previous similar message [ 8132.075464] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8133.298305] Lustre: server umount lustre-MDT0001 complete [ 8135.822626] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 8140.823961] Lustre: server umount lustre-OST0000 complete [ 8143.782192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8145.193869] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8146.235817] Lustre: 319968:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.105@tcp [ 8146.990500] Lustre: server umount MGS complete [ 8150.034852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8150.151373] LustreError: 320532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8150.202540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8151.529649] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8154.382244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8154.500243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8155.902159] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8158.984429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8161.259685] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8163.233578] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8178.285148] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 14 sec [ 8179.490149] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8180.038162] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8183.260781] Lustre: server umount lustre-OST0000 complete [ 8184.800245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8184.800828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8184.804393] Lustre: Skipped 1 previous similar message [ 8194.038225] Lustre: server umount lustre-MDT0000 complete [ 8195.040409] LustreError: 320538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8195.049671] LustreError: 320538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8195.293270] LustreError: 320518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346041 with bad export cookie 15713718621353912956 [ 8195.296690] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8195.297484] LustreError: 320518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8204.811607] Lustre: server umount lustre-MDT0001 complete [ 8208.456469] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8209.521344] Key type lgssc unregistered [ 8209.644370] LNet: 323735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8209.646729] LNetError: 323735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8209.653284] LNet: Removed LNI 192.168.201.105@tcp [ 8209.935143] Key type .llcrypt unregistered [ 8209.936359] Key type ._llcrypt unregistered [ 8216.091189] Key type ._llcrypt registered [ 8216.092492] Key type .llcrypt registered [ 8216.131840] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8222.713099] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8223.197473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8223.209518] alg: No test for adler32 (adler32-zlib) [ 8224.071920] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 8224.160100] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8225.743189] Key type lgssc registered [ 8226.131420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8237.802159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8237.806632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8238.892940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8238.905942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8238.942697] Lustre: lustre-MDT0000: new disk, initializing [ 8238.968885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8238.976762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8240.297198] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8244.419835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8244.462594] Lustre: 326833:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8244.484841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8244.487285] Lustre: Skipped 1 previous similar message [ 8244.521075] Lustre: lustre-MDT0001: new disk, initializing [ 8244.552539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8244.566550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8244.571240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8245.958181] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8249.198716] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8250.448560] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8252.692385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.791491] Lustre: lustre-OST0000: new disk, initializing [ 8252.793686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8252.796494] Lustre: 328576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8252.826156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8254.840656] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8255.414101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8255.416737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8255.447382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8257.979891] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8259.357995] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8259.427594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8260.722372] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8260.788842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8265.696543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8265.697473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8265.699409] LustreError: Skipped 1 previous similar message [ 8265.700036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8265.704117] Lustre: Skipped 1 previous similar message [ 8267.474255] Lustre: server umount lustre-OST0000 complete [ 8270.303640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8270.306040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.310350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8270.312213] Lustre: Skipped 1 previous similar message [ 8274.838829] Lustre: server umount lustre-MDT0000 complete [ 8276.205286] LustreError: 327612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346122 with bad export cookie 8516488021306296355 [ 8276.207672] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8276.210268] LustreError: 327612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8276.315113] Lustre: server umount lustre-MDT0001 complete [ 8279.384117] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:35:24 (1780346124) [ 8284.631406] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8287.807827] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8299.285253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8299.377993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8299.390120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8299.429717] Lustre: lustre-MDT0000: new disk, initializing [ 8299.455141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8299.462295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8300.704871] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8304.939706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8304.973572] Lustre: 333141:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8304.991044] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8304.992823] Lustre: Skipped 1 previous similar message [ 8305.024745] Lustre: lustre-MDT0001: new disk, initializing [ 8305.055598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8305.066368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8305.069645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8306.277609] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8309.227914] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8310.452306] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8312.667698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8312.765616] Lustre: 334881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8314.551542] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8316.531766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8316.535213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8316.561773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8317.464526] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8327.136750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8327.138457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8327.138731] LustreError: Skipped 1 previous similar message [ 8327.141504] Lustre: Skipped 1 previous similar message [ 8327.141972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8327.144958] Lustre: Skipped 2 previous similar messages [ 8331.990762] Lustre: server umount lustre-OST0000 complete [ 8333.793276] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8333.795492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8333.799832] Lustre: Skipped 1 previous similar message [ 8333.801876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8335.254428] Lustre: server umount lustre-MDT0000 complete [ 8336.413116] LustreError: 333134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346182 with bad export cookie 8516488021306297223 [ 8336.416083] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8336.416148] LustreError: 333134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8336.521286] Lustre: server umount lustre-MDT0001 complete [ 8339.620919] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8340.602817] Key type lgssc unregistered [ 8340.737430] LNet: 336900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8340.739215] LNetError: 336900:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8340.746271] LNet: Removed LNI 192.168.201.105@tcp [ 8341.029191] Key type .llcrypt unregistered [ 8341.030471] Key type ._llcrypt unregistered [ 8370.894842] Key type ._llcrypt registered [ 8370.895853] Key type .llcrypt registered [ 8370.930656] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8371.032209] Key type .llcrypt unregistered [ 8371.033469] Key type ._llcrypt unregistered [ 8396.771700] Key type ._llcrypt registered [ 8396.772636] Key type .llcrypt registered [ 8396.805528] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8396.889315] Key type .llcrypt unregistered [ 8396.890275] Key type ._llcrypt unregistered [ 8406.734886] Key type ._llcrypt registered [ 8406.735894] Key type .llcrypt registered [ 8406.772211] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8406.855211] Key type .llcrypt unregistered [ 8406.856125] Key type ._llcrypt unregistered [ 8416.758160] Key type ._llcrypt registered [ 8416.758968] Key type .llcrypt registered [ 8416.793908] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8416.885290] Key type .llcrypt unregistered [ 8416.886599] Key type ._llcrypt unregistered [ 8441.718920] Key type ._llcrypt registered [ 8441.719908] Key type .llcrypt registered [ 8441.751072] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8441.837371] Key type .llcrypt unregistered [ 8441.838595] Key type ._llcrypt unregistered [ 8452.884291] Key type ._llcrypt registered [ 8452.885229] Key type .llcrypt registered [ 8452.924229] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8453.028258] Key type .llcrypt unregistered [ 8453.029839] Key type ._llcrypt unregistered [ 8464.093205] Key type ._llcrypt registered [ 8464.094435] Key type .llcrypt registered [ 8464.138196] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8464.237277] Key type .llcrypt unregistered [ 8464.238219] Key type ._llcrypt unregistered [ 8474.841459] Key type ._llcrypt registered [ 8474.842464] Key type .llcrypt registered [ 8474.880513] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8474.971321] Key type .llcrypt unregistered [ 8474.972323] Key type ._llcrypt unregistered [ 8485.918150] Key type ._llcrypt registered [ 8485.919093] Key type .llcrypt registered [ 8485.954847] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8486.057249] Key type .llcrypt unregistered [ 8486.058278] Key type ._llcrypt unregistered [ 8496.716492] Key type ._llcrypt registered [ 8496.717718] Key type .llcrypt registered [ 8496.749309] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8496.835291] Key type .llcrypt unregistered [ 8496.836304] Key type ._llcrypt unregistered [ 8516.714745] Key type ._llcrypt registered [ 8516.715817] Key type .llcrypt registered [ 8516.749812] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8516.848413] Key type .llcrypt unregistered [ 8516.849845] Key type ._llcrypt unregistered [ 8528.060119] Key type ._llcrypt registered [ 8528.061136] Key type .llcrypt registered [ 8528.098732] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8528.201330] Key type .llcrypt unregistered [ 8528.202361] Key type ._llcrypt unregistered [ 8539.263552] Key type ._llcrypt registered [ 8539.264515] Key type .llcrypt registered [ 8539.311570] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8539.425412] Key type .llcrypt unregistered [ 8539.426566] Key type ._llcrypt unregistered [ 8557.747978] Key type ._llcrypt registered [ 8557.749171] Key type .llcrypt registered [ 8557.787903] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8564.201762] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8564.538088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8564.577075] alg: No test for adler32 (adler32-zlib) [ 8565.438875] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 8565.535485] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8567.119125] Key type lgssc registered [ 8567.529247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8578.886434] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8578.894223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.986307] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8579.998632] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8580.034850] Lustre: tst250fs-MDT0000: new disk, initializing [ 8580.059681] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8580.065592] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8581.327543] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8585.441986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8585.492548] Lustre: 356553:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8585.521458] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8585.523748] Lustre: Skipped 1 previous similar message [ 8585.567989] Lustre: tst250fs-MDT0001: new disk, initializing [ 8585.603198] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8585.617357] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8585.623500] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8587.002874] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8589.350685] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8591.578399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8591.678481] Lustre: tst250fs-OST0000: new disk, initializing [ 8591.680679] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8591.682995] Lustre: 358154:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8591.715048] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8591.782928] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8591.786261] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8591.836159] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8593.617382] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8597.719745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8597.786506] Lustre: tst250fs-OST0001: new disk, initializing [ 8597.789047] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8597.792903] Lustre: 359008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8597.826563] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8599.719534] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8600.640664] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8600.643638] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8600.675443] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8602.236907] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8615.905683] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8615.908354] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8615.909162] Lustre: Skipped 2 previous similar messages [ 8615.911866] Lustre: Skipped 1 previous similar message [ 8618.399225] Lustre: server umount tst250fs-MDT0000 complete [ 8619.634246] LustreError: 356546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346466 with bad export cookie 13186732731350143349 [ 8619.638753] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8619.755678] Lustre: server umount tst250fs-MDT0001 complete [ 8631.233884] Lustre: server umount tst250fs-OST0000 complete [ 8641.701804] Lustre: server umount tst250fs-OST0001 complete [ 8647.662879] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8650.027874] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8660.244846] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:41:45 (1780346505) [ 8660.900518] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8670.721622] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8676.661143] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8712.275365] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8408 sec ========== 16:42:36 (1780346556) [ 8715.660417] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:42:38 (1780346558) === [ 8721.792411] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:42:45 (1780346565) === [ 8780.834514] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8783.720239] Key type lgssc unregistered [ 8783.968556] LNet: 368026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8783.973462] LNetError: 368026:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8785.005318] LNet: Removed LNI 192.168.201.105@tcp [ 8785.662186] Key type .llcrypt unregistered [ 8785.664066] Key type ._llcrypt unregistered