[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 577252723 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003395] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.007722] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.013032] SELinux: Initializing. [ 0.014057] *** VALIDATE selinux *** [ 0.021602] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025645] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027092] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028098] *** VALIDATE tmpfs *** [ 0.029382] *** VALIDATE proc *** [ 0.030197] *** VALIDATE cgroup *** [ 0.031005] *** VALIDATE cgroup2 *** [ 0.032218] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033115] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035024] Spectre V2 : User space: Vulnerable [ 0.036004] Speculative Store Bypass: Vulnerable [ 0.038911] debug: unmapping init [mem 0xffffffff84c59000-0xffffffff84c60fff] [ 0.040195] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041531] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042014] ... version: 2 [ 0.042892] ... bit width: 48 [ 0.043007] ... generic registers: 4 [ 0.043873] ... value mask: 0000ffffffffffff [ 0.044008] ... max period: 00007fffffffffff [ 0.045007] ... fixed-purpose events: 3 [ 0.045893] ... event mask: 000000070000000f [ 0.046237] rcu: Hierarchical SRCU implementation. [ 0.048244] smp: Bringing up secondary CPUs ... [ 0.049492] x86: Booting SMP configuration: [ 0.050013] .... node #0, CPUs: #1 #2 #3 [ 0.058175] smp: Brought up 1 node, 4 CPUs [ 0.060008] smpboot: Max logical packages: 1 [ 0.061009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.089000] node 0 deferred pages initialised in 25ms [ 0.090026] devtmpfs: initialized [ 0.091179] x86/mm: Memory block size: 128MB [ 0.092000] gcov: version magic: 0x41383552 [ 0.094055] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.095061] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.097229] pinctrl core: initialized pinctrl subsystem [ 0.099141] [ 0.099485] ************************************************************* [ 0.101009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.103009] ** ** [ 0.105008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.106007] ** ** [ 0.108009] ** This means that this kernel is built to expose internal ** [ 0.110008] ** IOMMU data structures, which may compromise security on ** [ 0.112008] ** your system. ** [ 0.113009] ** ** [ 0.115020] ** If you see this message and you are not debugging the ** [ 0.116007] ** kernel, report this immediately to your vendor! ** [ 0.117000] ** ** [ 0.117000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.119008] ************************************************************* [ 0.120631] NET: Registered protocol family 16 [ 0.122464] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.124141] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.126045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.129086] cpuidle: using governor menu [ 0.130000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.131475] PCI: Using configuration type 1 for base access [ 0.133204] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.138307] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.139000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.140060] cryptd: max_cpu_qlen set to 1000 [ 0.142179] ACPI: Added _OSI(Module Device) [ 0.143009] ACPI: Added _OSI(Processor Device) [ 0.144073] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.145015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.150135] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.156055] ACPI: Interpreter enabled [ 0.157044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.158008] ACPI: Using IOAPIC for interrupt routing [ 0.160070] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.162316] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.173915] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.175024] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.177011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.180076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.184333] acpiphp: Slot [2] registered [ 0.185086] acpiphp: Slot [5] registered [ 0.186084] acpiphp: Slot [6] registered [ 0.187082] acpiphp: Slot [7] registered [ 0.188093] acpiphp: Slot [8] registered [ 0.189112] acpiphp: Slot [9] registered [ 0.190163] acpiphp: Slot [10] registered [ 0.192069] acpiphp: Slot [3] registered [ 0.193097] acpiphp: Slot [4] registered [ 0.194103] acpiphp: Slot [11] registered [ 0.195070] acpiphp: Slot [12] registered [ 0.196071] acpiphp: Slot [13] registered [ 0.197070] acpiphp: Slot [14] registered [ 0.198089] acpiphp: Slot [15] registered [ 0.199136] acpiphp: Slot [16] registered [ 0.201059] acpiphp: Slot [17] registered [ 0.202072] acpiphp: Slot [18] registered [ 0.203109] acpiphp: Slot [19] registered [ 0.204064] acpiphp: Slot [20] registered [ 0.205097] acpiphp: Slot [21] registered [ 0.206058] acpiphp: Slot [22] registered [ 0.207058] acpiphp: Slot [23] registered [ 0.208075] acpiphp: Slot [24] registered [ 0.209068] acpiphp: Slot [25] registered [ 0.210072] acpiphp: Slot [26] registered [ 0.211230] acpiphp: Slot [27] registered [ 0.213217] acpiphp: Slot [28] registered [ 0.214126] acpiphp: Slot [29] registered [ 0.215089] acpiphp: Slot [30] registered [ 0.216182] acpiphp: Slot [31] registered [ 0.218054] PCI host bridge to bus 0000:00 [ 0.220024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.222036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.224026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.226017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.229020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.231038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.233226] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.235864] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.239135] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.248706] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.253048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.256019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.258014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.261020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.263488] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.266660] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.269039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.271333] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.275014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.286775] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.290012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.294135] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.301014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.307014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.322021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.333080] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.340015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.345014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.358014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.364899] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.375014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.381015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.397018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.416297] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.422013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.429013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.444020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.452629] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.459014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.467013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.481014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.490099] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.497014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.504016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.519029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.534724] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.537691] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.540295] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.542423] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.545227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.549044] iommu: Default domain type: Passthrough [ 0.550352] SCSI subsystem initialized [ 0.552118] ACPI: bus type USB registered [ 0.553077] usbcore: registered new interface driver usbfs [ 0.555104] usbcore: registered new interface driver hub [ 0.556071] usbcore: registered new device driver usb [ 0.558136] pps_core: LinuxPPS API ver. 1 registered [ 0.560009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.562048] PTP clock support registered [ 0.564163] EDAC MC: Ver: 3.0.0 [ 0.566148] PCI: Using ACPI for IRQ routing [ 0.567000] NetLabel: Initializing [ 0.569009] NetLabel: domain hash size = 128 [ 0.570006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.572073] NetLabel: unlabeled traffic allowed by default [ 0.574007] vgaarb: loaded [ 0.575555] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.577010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.583244] clocksource: Switched to clocksource kvm-clock [ 0.694892] VFS: Disk quotas dquot_6.6.0 [ 0.696306] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.698498] *** VALIDATE ramfs *** [ 0.699470] *** VALIDATE hugetlbfs *** [ 0.700713] pnp: PnP ACPI init [ 0.702955] pnp: PnP ACPI: found 6 devices [ 0.717875] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.720678] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.722438] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.724239] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.726052] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.727876] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.730083] NET: Registered protocol family 2 [ 0.732058] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.736421] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.738938] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.743276] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.746042] TCP: Hash tables configured (established 65536 bind 65536) [ 0.748316] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.750727] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.752878] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.755175] NET: Registered protocol family 1 [ 0.757772] RPC: Registered named UNIX socket transport module. [ 0.759549] RPC: Registered udp transport module. [ 0.761245] RPC: Registered tcp transport module. [ 0.762425] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.764469] NET: Registered protocol family 44 [ 0.765877] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.767658] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.769490] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.771483] PCI: CLS 0 bytes, default 64 [ 0.772883] Unpacking initramfs... [ 2.428696] debug: unmapping init [mem 0xffff8ce83cc54000-0xffff8ce83ffbffff] [ 2.432919] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.435128] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.438215] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.012908] Initialise system trusted keyrings [ 3.014504] Key type blacklist registered [ 3.016447] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.026075] zbud: loaded [ 3.029141] *** VALIDATE nfs *** [ 3.030326] *** VALIDATE nfs4 *** [ 3.031810] pstore: using deflate compression [ 3.035720] Platform Keyring initialized [ 3.149317] NET: Registered protocol family 38 [ 3.151450] Key type asymmetric registered [ 3.153284] Asymmetric key parser 'x509' registered [ 3.155329] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.158597] io scheduler mq-deadline registered [ 3.160412] io scheduler kyber registered [ 3.162234] io scheduler bfq registered [ 3.164101] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.167289] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.170376] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.173385] ACPI: Power Button [PWRF] [ 3.179241] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.186748] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.202369] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.210085] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.225457] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.254042] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.283807] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.289657] Non-volatile memory driver v1.3 [ 3.291363] Linux agpgart interface v0.103 [ 3.318695] virtio_blk virtio1: [vda] 145784 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.321184] vda: detected capacity change from 0 to 74641408 [ 3.334173] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.336576] vdb: detected capacity change from 0 to 1073741824 [ 3.350324] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.353262] vdc: detected capacity change from 0 to 2621440000 [ 3.368843] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.371687] vdd: detected capacity change from 0 to 2621440000 [ 3.384515] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.386854] vde: detected capacity change from 0 to 4294967296 [ 3.401441] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.404449] vdf: detected capacity change from 0 to 4294967296 [ 3.412279] libphy: Fixed MDIO Bus: probed [ 3.420208] usbcore: registered new interface driver usbserial_generic [ 3.428092] usbserial: USB Serial support registered for generic [ 3.479912] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.487176] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.490620] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.494259] mousedev: PS/2 mouse device common for all mice [ 3.498406] rtc_cmos 00:05: RTC can wake from S4 [ 3.501537] rtc_cmos 00:05: registered as rtc0 [ 3.503375] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.506965] intel_pstate: CPU model not supported [ 3.507995] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.513483] hid: raw HID events driver (C) Jiri Kosina [ 3.515590] usbcore: registered new interface driver usbhid [ 3.517652] usbhid: USB HID core driver [ 3.519190] drop_monitor: Initializing network drop monitor service [ 3.521618] Initializing XFRM netlink socket [ 3.522475] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.524345] NET: Registered protocol family 10 [ 3.529904] Segment Routing with IPv6 [ 3.531125] NET: Registered protocol family 17 [ 3.532739] mpls_gso: MPLS GSO support [ 3.534682] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.541668] RAS: Correctable Errors collector initialized. [ 3.543439] AVX version of gcm_enc/dec engaged. [ 3.544651] AES CTR mode by8 optimization enabled [ 3.625950] sched_clock: Marking stable (3625923876, 0)->(4684384229, -1058460353) [ 3.629291] registered taskstats version 1 [ 3.631855] Loading compiled-in X.509 certificates [ 3.634086] zswap: loaded using pool lzo/zbud [ 3.660992] Key type big_key registered [ 3.673623] Key type encrypted registered [ 3.676546] ima: No TPM chip found, activating TPM-bypass! [ 3.678648] ima: Allocated hash algorithm: sha1 [ 3.680441] ima: No architecture policies found [ 3.682534] evm: Initialising EVM extended attributes: [ 3.685143] evm: security.selinux [ 3.686569] evm: security.ima [ 3.688080] evm: security.capability [ 3.689681] evm: HMAC attrs: 0x1 [ 3.692605] rtc_cmos 00:05: setting system clock to 2026-07-15 13:25:28 UTC (1784121928) [ 3.700479] debug: unmapping init [mem 0xffffffff85c03000-0xffffffff85dfffff] [ 3.704304] debug: unmapping init [mem 0xffffffff84982000-0xffffffff84c58fff] [ 3.716192] Write protecting the kernel read-only data: 28672k [ 3.720439] debug: unmapping init [mem 0xffffffff83003000-0xffffffff831fffff] [ 3.723393] debug: unmapping init [mem 0xffffffff83914000-0xffffffff839fffff] [ 3.767214] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.780479] systemd[1]: Detected virtualization kvm. [ 3.783571] systemd[1]: Detected architecture x86-64. [ 3.787914] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.818908] systemd[1]: No hostname configured. [ 3.822152] systemd[1]: Set hostname to . [ 3.825335] random: systemd: uninitialized urandom read (16 bytes read) [ 3.829111] systemd[1]: Initializing machine ID from random generator. [ 4.170970] random: systemd: uninitialized urandom read (16 bytes read) [ 4.174503] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.180877] random: systemd: uninitialized urandom read (16 bytes read) [ 4.184370] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.192418] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.719894] device-mapper: uevent: version 1.0.3 [ 5.721776] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.868791] random: fast init done [ 6.895208] virtio_net virtio0 ens2: renamed from eth0 [ 7.104164] scsi host0: ata_piix [ 7.351576] scsi host1: ata_piix [ 7.355395] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.366316] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.208230] random: crng init done [ 12.215020] random: 7 urandom warning(s) missed due to ratelimiting [ 13.975443] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.915079] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.160391] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.539589] SELinux: Disabled at runtime. [ 18.657381] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.671258] systemd[1]: Detected virtualization kvm. [ 18.673519] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.644126] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.649362] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.683078] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.689248] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.695537] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.705802] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.713856] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of[ 19.841320] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.596525] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.417400] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.546925] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.797545] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.841881] EDAC sbridge: Ver: 1.1.2 [ 25.074339] Key type dns_resolver registered [ 25.526980] NFS: Registering the id_resolver key type [ 25.532263] Key type id_resolver registered [ 25.536400] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ 26.806168] hrtimer: interrupt took 3554762 ns [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg131-server login: [ 66.785990] libcfs: loading out-of-tree module taints kernel. [ 66.807922] Key type ._llcrypt registered [ 66.809182] Key type .llcrypt registered [ 66.850829] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 74.678369] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 75.177775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.184653] alg: No test for adler32 (adler32-zlib) [ 76.199542] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 76.538266] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 78.160635] Key type lgssc registered [ 78.735925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.573773] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 104.188735] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 110.354239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 110.375711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 111.489422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 111.504491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 111.544159] Lustre: lustre-MDT0000: new disk, initializing [ 111.579606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 111.588744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 113.339019] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 119.663217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 119.710632] Lustre: 6485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 119.724617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 119.727576] Lustre: Skipped 1 previous similar message [ 119.767426] Lustre: lustre-MDT0001: new disk, initializing [ 119.790513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 119.801493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 119.805756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 121.416932] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 124.078060] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 128.635483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 128.759702] Lustre: lustre-OST0000: new disk, initializing [ 128.762626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 128.765327] Lustre: 8424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 128.792178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 131.546380] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 133.634238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 133.645532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 133.677143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 138.574587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 138.630686] Lustre: lustre-OST0001: new disk, initializing [ 138.633170] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 138.636639] Lustre: 9495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 138.673856] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 139.890541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 139.894331] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 139.914495] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 141.799516] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 149.645821] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 152.602306] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 155.381299] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing check_logdir /tmp/testlogs/ [ 157.471437] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing yml_node [ 159.436560] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 160.558833] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 161.765846] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 162.571872] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jul 15 09:28:06 EDT 2026 [ 169.864866] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 170.460601] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 186.339567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 186.340220] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 186.344110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 186.348572] Lustre: Skipped 3 previous similar messages [ 186.353450] Lustre: Skipped 3 previous similar messages [ 188.451362] Lustre: server umount lustre-MDT0000 complete [ 190.062893] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122114 with bad export cookie 7514835158809482130 [ 190.064052] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 190.067270] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 191.456677] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 191.457000] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 191.461292] Lustre: Skipped 1 previous similar message [ 195.488714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 195.491765] Lustre: Skipped 2 previous similar messages [ 196.271879] Lustre: server umount lustre-MDT0001 complete [ 203.779097] Lustre: server umount lustre-OST0000 complete [ 211.158215] Lustre: server umount lustre-OST0001 complete [ 213.834960] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 217.574585] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 231.730979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.834681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 231.848786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 231.885381] Lustre: lustre-MDT0000: new disk, initializing [ 231.919880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.934638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 233.702966] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 238.668738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.716361] Lustre: 15037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 238.845794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 240.322237] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 244.591981] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 247.250578] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 249.845151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.964287] Lustre: lustre-OST0000: new disk, initializing [ 249.965697] Lustre: Skipped 1 previous similar message [ 249.967955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 249.970824] Lustre: Skipped 2 previous similar messages [ 249.973257] Lustre: 17005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.003839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.006270] Lustre: Skipped 1 previous similar message [ 251.707031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 251.709914] Lustre: Skipped 1 previous similar message [ 251.712165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 251.751960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.210690] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 256.373114] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 257.871825] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 257.953961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 259.551578] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 259.649127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 262.114033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 262.114855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.116798] LustreError: Skipped 1 previous similar message [ 262.121952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 262.124862] Lustre: Skipped 1 previous similar message [ 266.588912] Lustre: server umount lustre-OST0000 complete [ 269.793101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 269.796956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.802043] Lustre: Skipped 1 previous similar message [ 269.804678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 274.216533] Lustre: server umount lustre-MDT0000 complete [ 275.900616] LustreError: 15029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122200 with bad export cookie 7514835158809483663 [ 275.905687] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 275.907676] LustreError: 15029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 276.075381] Lustre: server umount lustre-MDT0001 complete [ 278.723209] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:30:03 (1784122203) [ 285.638329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 285.820611] LustreError: 19874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 285.829706] LustreError: 19874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 285.847136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 287.381975] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 290.681287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 292.336645] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 295.002083] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 297.242905] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 298.074731] Lustre: Failing over lustre-MDT0000 [ 298.169047] Lustre: server umount lustre-MDT0000 complete [ 301.031211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 301.035043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.040637] Lustre: Skipped 1 previous similar message [ 301.042950] LustreError: 19893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.049380] LustreError: 19874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.057099] LustreError: 19874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 307.169300] LustreError: 19875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.177900] LustreError: 19875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 312.264885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 312.289851] LustreError: 19893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.297609] LustreError: 19893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 312.340688] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.219714] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 316.580693] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 316.580919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 316.595389] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 318.934107] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 322.018646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.018937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 322.026323] Lustre: Skipped 2 previous similar messages [ 322.026962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.026966] Lustre: Skipped 2 previous similar messages [ 327.459049] Lustre: server umount lustre-MDT0000 complete [ 329.174172] LustreError: 19856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122253 with bad export cookie 7514835158809485084 [ 329.177323] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 329.178695] LustreError: 19856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 329.303690] Lustre: server umount lustre-MDT0001 complete [ 332.614910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.799966] LustreError: 23206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.829380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 332.832421] Lustre: Skipped 2 previous similar messages [ 334.427967] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 337.698035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 339.361239] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 342.271121] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 344.478196] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 345.375411] Lustre: Failing over lustre-MDT0000 [ 345.477777] Lustre: server umount lustre-MDT0000 complete [ 347.425627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.430725] Lustre: Skipped 1 previous similar message [ 352.737287] LustreError: 23949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.744503] LustreError: 23949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 359.059307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.110781] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.863761] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 362.274704] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 362.280484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 362.282363] Lustre: Skipped 1 previous similar message [ 362.288806] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 365.238209] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 367.319552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.323958] Lustre: Skipped 4 previous similar messages [ 373.469283] Lustre: server umount lustre-MDT0000 complete [ 374.874387] LustreError: 23188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122299 with bad export cookie 7514835158809486022 [ 374.877227] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.878618] LustreError: 23188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 375.004228] Lustre: server umount lustre-MDT0001 complete [ 378.169300] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:31:42 (1784122302) [ 384.863579] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 389.269465] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 405.942717] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 410.347036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.441957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 410.464471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 410.506164] Lustre: lustre-MDT0000: new disk, initializing [ 410.540013] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.543627] Lustre: Skipped 2 previous similar messages [ 410.550925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 412.130929] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 416.636422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.681782] Lustre: 29568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 416.794209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.350698] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 422.482245] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 424.550423] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 427.303338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 427.430374] Lustre: 31535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 429.449325] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 429.474744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 429.481536] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 429.839665] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 434.122278] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 437.090915] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 444.898278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 444.899229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.901382] LustreError: Skipped 1 previous similar message [ 444.906014] Lustre: Skipped 2 previous similar messages [ 444.907229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 444.911987] Lustre: Skipped 3 previous similar messages [ 448.561988] Lustre: server umount lustre-OST0000 complete [ 456.345047] Lustre: server umount lustre-MDT0000 complete [ 457.637181] LustreError: 31534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122382 with bad export cookie 7514835158809486421 [ 457.639918] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 457.641568] LustreError: 31534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 457.826176] Lustre: server umount lustre-MDT0001 complete [ 461.956023] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 463.072178] Key type lgssc unregistered [ 463.200425] LNet: 33487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.203091] LNetError: 33487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 463.210448] LNet: Removed LNI 192.168.201.131@tcp [ 463.570183] Key type .llcrypt unregistered [ 463.571888] Key type ._llcrypt unregistered [ 473.226652] Key type ._llcrypt registered [ 473.227978] Key type .llcrypt registered [ 473.265500] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 473.716918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 473.731927] alg: No test for adler32 (adler32-zlib) [ 474.596855] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 474.699192] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 476.304145] Key type lgssc registered [ 476.760475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 479.614613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 479.621197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.774123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.447561] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 485.699085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.840722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 487.246119] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 489.779863] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 491.757478] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.203357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.334732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.402082] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 496.432364] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 498.856786] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 515.021837] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:33:59 (1784122439) [ 515.812748] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 516.675716] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 519.176734] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:34:03 (1784122443) [ 522.555402] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:34:06 (1784122446) [ 522.947536] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 526.026440] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 526.029413] Lustre: Skipped 2 previous similar messages [ 526.698415] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:34:11 (1784122451) [ 530.426890] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 530.429789] Lustre: Skipped 7 previous similar messages [ 535.522971] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:34:19 (1784122459) [ 536.712804] Lustre: *** cfs_fail_loc=131b, val=2*** [ 538.105565] Lustre: *** cfs_fail_loc=131b, val=2*** [ 541.437311] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:34:25 (1784122465) [ 542.270337] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 542.272498] Lustre: Skipped 6 previous similar messages [ 547.080679] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:34:31 (1784122471) [ 551.907172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 551.907485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.908173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 551.910889] LustreError: Skipped 1 previous similar message [ 551.916236] Lustre: Skipped 1 previous similar message [ 557.028170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 557.031282] Lustre: Skipped 2 previous similar messages [ 557.274886] Lustre: server umount lustre-OST0000 complete [ 562.145547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.146189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 562.149873] Lustre: Skipped 1 previous similar message [ 564.823720] Lustre: server umount lustre-MDT0000 complete [ 566.087224] LustreError: 34860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122490 with bad export cookie 7255794583535908716 [ 566.089910] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.091102] LustreError: 34860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 566.204624] Lustre: server umount lustre-MDT0001 complete [ 569.737153] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 570.824770] Key type lgssc unregistered [ 570.955703] LNet: 42717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.959348] LNetError: 42717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 570.969387] LNet: Removed LNI 192.168.201.131@tcp [ 571.322253] Key type .llcrypt unregistered [ 571.323134] Key type ._llcrypt unregistered [ 577.560140] Key type ._llcrypt registered [ 577.561705] Key type .llcrypt registered [ 577.603326] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 584.355390] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 584.798545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 584.810416] alg: No test for adler32 (adler32-zlib) [ 585.665628] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 585.762196] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 587.352147] Key type lgssc registered [ 587.760723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.512468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 601.541738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.674631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 602.702444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 602.747159] Lustre: lustre-MDT0000: new disk, initializing [ 602.780430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.790910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 604.596783] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 610.630145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.698338] Lustre: 45844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 610.728915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 610.735176] Lustre: Skipped 1 previous similar message [ 610.783129] Lustre: lustre-MDT0001: new disk, initializing [ 610.857987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 610.877364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 610.884976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 612.962677] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 617.796339] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 620.009503] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 622.655549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 622.783655] Lustre: lustre-OST0000: new disk, initializing [ 622.786120] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 622.788982] Lustre: 47812:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 622.825763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.043402] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 628.228412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 628.233585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 628.272654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 629.125798] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 630.683171] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 630.762936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 632.364909] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 632.450805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 633.314976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.315964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 633.316525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 633.319626] Lustre: Skipped 1 previous similar message [ 633.322764] LustreError: Skipped 1 previous similar message [ 635.237633] Lustre: server umount lustre-OST0000 complete [ 638.433792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.434611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.438973] Lustre: Skipped 1 previous similar message [ 638.444325] Lustre: Skipped 2 previous similar messages [ 642.843111] Lustre: server umount lustre-MDT0000 complete [ 644.216709] LustreError: 47811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122569 with bad export cookie 14402312354909838060 [ 644.219388] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.222176] LustreError: 47811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 644.342853] Lustre: server umount lustre-MDT0001 complete [ 648.027236] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:36:12 (1784122572) [ 650.511476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.674912] LustreError: 50267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.681566] LustreError: 50267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 650.708017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 652.165228] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 655.392131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.565260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 657.086929] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 659.716331] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 661.864460] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 664.678561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 664.826354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.021226] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 669.806719] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 682.440436] LustreError: 52818:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 682.826297] Lustre: Modifying parameter general.timeout=1 in log params [ 683.830242] Lustre: Modifying parameter general.timeout=4 in log params [ 683.832408] Lustre: Skipped 2 previous similar messages [ 686.127742] Lustre: Modifying parameter general.timeout=11 in log params [ 686.129925] Lustre: Skipped 6 previous similar messages [ 690.325857] Lustre: Modifying parameter general.timeout=23 in log params [ 690.329441] Lustre: Skipped 11 previous similar messages [ 698.455357] Lustre: Modifying parameter general.timeout=47 in log params [ 698.457812] Lustre: Skipped 23 previous similar messages [ 714.710292] Lustre: Modifying parameter general.timeout=93 in log params [ 714.712223] Lustre: Skipped 45 previous similar messages [ 731.879979] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:37:36 (1784122656) [ 733.030640] Lustre: 59718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 734.597835] Lustre: 59914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 734.603105] Lustre: 59914:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 736.931172] Lustre: 60208:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 736.936194] Lustre: 60208:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 747.700386] Lustre: 61560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 747.705803] LustreError: 61560:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 748.129050] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_policies= in log params [ 748.131547] Lustre: Skipped 57 previous similar messages [ 749.462248] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:37:53 (1784122673) [ 756.194874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.196563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.201309] Lustre: Skipped 2 previous similar messages [ 756.205158] Lustre: Skipped 2 previous similar messages [ 760.375817] Lustre: server umount lustre-MDT0000 complete [ 761.313173] LustreError: 50267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 761.319108] LustreError: 50267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 761.934056] LustreError: 50247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122686 with bad export cookie 14402312354909838942 [ 761.936492] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.939605] LustreError: 50247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 762.115688] Lustre: server umount lustre-MDT0001 complete [ 774.152573] Lustre: server umount lustre-OST0000 complete [ 785.846823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.006307] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 786.146418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 787.947119] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 791.422149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.470785] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 791.482819] Lustre: 64085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 793.117422] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 795.906395] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 797.923357] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 800.407773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 800.505321] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 803.026678] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 806.147593] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 811.978685] Lustre: 66653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 812.658820] Lustre: Modifying parameter general.jobid_var=disable in log params [ 812.661864] Lustre: Skipped 4 previous similar messages [ 816.609774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 816.610594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 816.613247] LustreError: Skipped 1 previous similar message [ 816.613912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 816.619463] Lustre: Skipped 1 previous similar message [ 820.186497] Lustre: server umount lustre-OST0000 complete [ 822.245361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 822.251430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 822.259027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 822.262599] Lustre: Skipped 1 previous similar message [ 831.969199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 831.971710] Lustre: Skipped 3 previous similar messages [ 836.064167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 836.131733] Lustre: server umount lustre-MDT0000 complete [ 837.089231] LustreError: 64097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 837.096243] LustreError: 64097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 837.498283] LustreError: 64078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122762 with bad export cookie 14402312354909841259 [ 837.501778] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.503137] LustreError: 64078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 837.633043] Lustre: server umount lustre-MDT0001 complete [ 841.971453] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 843.160294] Key type lgssc unregistered [ 843.306336] LNet: 67864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.309426] LNetError: 67864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.319339] LNet: Removed LNI 192.168.201.131@tcp [ 843.684147] Key type .llcrypt unregistered [ 843.685948] Key type ._llcrypt unregistered [ 854.260412] Key type ._llcrypt registered [ 854.261791] Key type .llcrypt registered [ 854.303034] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 854.656817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 854.691758] alg: No test for adler32 (adler32-zlib) [ 855.582299] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 855.699790] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 857.288755] Key type lgssc registered [ 857.727460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 860.981188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 860.992517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 862.295777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 864.081545] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 867.465076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.694950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 869.348320] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 872.651080] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 875.255665] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 878.249857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 878.498903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.868947] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 883.944671] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 897.199788] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 913.380250] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.381512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.384279] Lustre: Skipped 2 previous similar messages [ 913.386181] Lustre: Skipped 1 previous similar message [ 916.442858] Lustre: server umount lustre-MDT0000 complete [ 918.069397] LustreError: 69233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122842 with bad export cookie 12813026085233602897 [ 918.072501] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.074966] LustreError: 69233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 918.239428] Lustre: server umount lustre-MDT0001 complete [ 930.012299] Lustre: server umount lustre-OST0000 complete [ 940.580672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.710562] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 940.724538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 940.826892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.382588] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 945.985158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.050711] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 946.064991] Lustre: 74166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 946.274693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 947.953522] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 950.820357] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 952.881051] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 955.776138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 955.861697] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 955.905260] LustreError: 76011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 955.913205] LustreError: 76011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 956.055970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.677330] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 961.971850] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 962.665421] Lustre: 76736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 963.477576] Lustre: Modifying parameter general.jobid_var=disable in log params [ 966.628893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 966.629176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 966.629745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 966.629749] Lustre: Skipped 1 previous similar message [ 966.636532] Lustre: Skipped 1 previous similar message [ 971.218259] Lustre: server umount lustre-OST0000 complete [ 976.867223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 976.867792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.868346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 976.868350] Lustre: Skipped 1 previous similar message [ 976.871367] LustreError: Skipped 1 previous similar message [ 976.875911] Lustre: Skipped 1 previous similar message [ 978.794640] Lustre: server umount lustre-MDT0000 complete [ 980.270275] LustreError: 74158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122905 with bad export cookie 12813026085233603730 [ 980.273557] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.274549] LustreError: 74158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 980.414702] Lustre: server umount lustre-MDT0001 complete [ 984.525570] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 985.686784] Key type lgssc unregistered [ 985.838753] LNet: 77944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.845581] LNetError: 77944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.855484] LNet: Removed LNI 192.168.201.131@tcp [ 986.210683] Key type .llcrypt unregistered [ 986.212396] Key type ._llcrypt unregistered [ 996.677850] Key type ._llcrypt registered [ 996.679271] Key type .llcrypt registered [ 996.717925] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 997.063314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.075466] alg: No test for adler32 (adler32-zlib) [ 997.950815] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 998.052618] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 999.640275] Key type lgssc registered [ 1000.045914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1003.061771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1003.068735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.242421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.853324] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1009.636432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.867413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1011.455824] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1014.326927] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1016.464243] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1019.179960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1019.450828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.002474] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1025.074092] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1032.848908] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1033.574083] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1033.576770] Lustre: Skipped 1 previous similar message [ 1034.600084] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1034.602827] Lustre: Skipped 2 previous similar messages [ 1036.901052] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1036.903793] Lustre: Skipped 6 previous similar messages [ 1041.034517] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1041.037754] Lustre: Skipped 11 previous similar messages [ 1049.298846] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1049.304780] Lustre: Skipped 23 previous similar messages [ 1065.407273] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1065.410022] Lustre: Skipped 44 previous similar messages [ 1070.568965] Lustre: 87404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1071.583198] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:43:15 (1784122995) [ 1072.172223] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1072.828245] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:43:17 (1784122997) [ 1075.682844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.684101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1075.684171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1075.688635] Lustre: Skipped 1 previous similar message [ 1080.024718] Lustre: server umount lustre-OST0000 complete [ 1085.921677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.922212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1085.926307] Lustre: Skipped 1 previous similar message [ 1085.928948] Lustre: Skipped 1 previous similar message [ 1087.641590] Lustre: server umount lustre-MDT0000 complete [ 1088.954265] LustreError: 79318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123013 with bad export cookie 13170841782823381500 [ 1088.957793] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1088.962117] LustreError: 79318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1089.080219] Lustre: server umount lustre-MDT0001 complete [ 1093.096665] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1094.217525] Key type lgssc unregistered [ 1094.352436] LNet: 88649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.355684] LNetError: 88649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1094.365685] LNet: Removed LNI 192.168.201.131@tcp [ 1094.702357] Key type .llcrypt unregistered [ 1094.703706] Key type ._llcrypt unregistered [ 1096.249280] Key type ._llcrypt registered [ 1096.250638] Key type .llcrypt registered [ 1096.292278] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1097.778486] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules [ 1098.114352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1098.149066] alg: No test for adler32 (adler32-zlib) [ 1099.010293] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1099.100911] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1100.688176] Key type lgssc registered [ 1101.146091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1101.622558] LustreError: 89803:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1109.761892] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1112.955151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1122.818148] LustreError: 89803:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1122.828134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.990982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.476392] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1128.448458] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:44:12 (1784123052) [ 1130.092747] LustreError: 90859:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1130.097203] LustreError: 90859:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1130.101274] LustreError: 90859:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1130.230022] Lustre: server umount lustre-MDT0000 complete [ 1135.188367] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1136.305312] Key type lgssc unregistered [ 1136.451471] LNet: 92351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.454883] LNetError: 92351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.462354] LNet: Removed LNI 192.168.201.131@tcp [ 1136.808226] Key type .llcrypt unregistered [ 1136.809632] Key type ._llcrypt unregistered [ 1144.229495] Key type ._llcrypt registered [ 1144.230769] Key type .llcrypt registered [ 1144.273153] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1144.689882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1144.697931] alg: No test for adler32 (adler32-zlib) [ 1145.546419] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1145.646458] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1147.232145] Key type lgssc registered [ 1147.606617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1150.312896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1150.319347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.479464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1152.928314] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1155.987922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1156.215849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1157.627029] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1160.157170] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1161.991624] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1164.344725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1164.561710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1166.578491] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1169.115371] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1201.527542] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:45:25 (1784123125) [ 1202.091031] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1202.676410] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:45:27 (1784123127) [ 1207.267751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1207.270990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.276544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.778377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.783710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.785729] Lustre: Skipped 1 previous similar message [ 1209.816957] Lustre: server umount lustre-MDT0000 complete [ 1211.066286] LustreError: 93525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123135 with bad export cookie 5601932111449227053 [ 1211.067940] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1211.071771] LustreError: 93525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1211.182618] Lustre: server umount lustre-MDT0001 complete [ 1222.791449] Lustre: server umount lustre-OST0000 complete [ 1226.873458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1227.032916] LustreError: 97833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1227.039419] LustreError: 97833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1227.090890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.527833] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1231.656108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.922906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1233.350559] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1235.987978] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1237.920650] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1242.318172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1242.393734] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1242.396432] LustreError: 97822:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1242.404976] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1242.408934] LustreError: 99915:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1242.411734] LustreError: 99915:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1242.414789] LustreError: 99915:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1242.442928] Lustre: server umount lustre-OST0000 complete [ 1242.444294] LustreError: 99915:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1245.374457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1245.440827] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1245.443365] LustreError: 97821:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1245.452451] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1245.456064] LustreError: 100379:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1245.458419] LustreError: 100379:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1245.460514] LustreError: 100379:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1245.483973] Lustre: server umount lustre-OST0000 complete [ 1245.485339] LustreError: 100379:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1248.857598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1248.948944] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1248.951251] Lustre: Found index 0 for lustre-OST0000, updating log [ 1248.954176] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1248.979782] Lustre: lustre-OST0000: new disk, initializing [ 1248.982554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1248.986583] Lustre: 100908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1249.088882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.157611] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1252.470716] Lustre: Failing over lustre-OST0000 [ 1252.553549] Lustre: server umount lustre-OST0000 complete [ 1256.993756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.994492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.997983] Lustre: Skipped 2 previous similar messages [ 1262.052145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1262.054266] Lustre: Skipped 2 previous similar messages [ 1267.168953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1267.171424] Lustre: Skipped 1 previous similar message [ 1268.192111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1268.246187] Lustre: server umount lustre-MDT0000 complete [ 1269.468335] LustreError: 97813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123194 with bad export cookie 5601932111449229783 [ 1269.471311] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1269.473063] LustreError: 97813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1269.572459] Lustre: server umount lustre-MDT0001 complete [ 1272.401688] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:46:36 (1784123196) [ 1277.496533] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1281.016354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.147267] LustreError: 103251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.169179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.527990] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1285.382510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.887464] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1287.888866] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1290.073526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1290.286469] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1291.297952] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1291.311898] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1291.312248] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1291.318680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1292.314308] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1295.301444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1295.343414] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1295.384296] Lustre: lustre-OST0001: new disk, initializing [ 1295.386603] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1295.390063] Lustre: 105568:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1297.608696] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1300.891495] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1301.003402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1301.007808] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1301.042299] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1306.694052] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1308.711960] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.31@tcp (stopping) [ 1310.688780] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1310.690978] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.129827] Lustre: server umount lustre-MDT0001 complete [ 1316.321225] LustreError: 103252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1316.327907] LustreError: 103252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1317.155571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.185490] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1317.187196] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1317.199236] Lustre: 103240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1317.425411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1317.427503] Lustre: Skipped 3 previous similar messages [ 1318.714356] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1318.759746] LustreError: 107333:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1318.763195] LustreError: 107333:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1318.767584] LustreError: 103243:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1318.797932] LustreError: 107346:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.131@tcp: cfg command failed: rc = -17 [ 1318.800489] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1318.800489] [ 1321.725461] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:47:26 (1784123246) [ 1322.470387] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1326.893711] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1332.681536] Lustre: 109172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1340.623083] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1341.785030] Lustre: 110304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1389.024692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1389.025067] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.025327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1389.025331] Lustre: Skipped 5 previous similar messages [ 1389.034262] Lustre: Skipped 5 previous similar messages [ 1393.814288] Lustre: server umount lustre-MDT0000 complete [ 1394.145513] LustreError: 104925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1394.150595] LustreError: 104925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1395.054757] LustreError: 103232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123319 with bad export cookie 5601932111449230700 [ 1395.056392] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1395.059970] LustreError: 103232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1395.170580] Lustre: server umount lustre-MDT0001 complete [ 1406.466803] Lustre: server umount lustre-OST0000 complete [ 1417.940918] Lustre: server umount lustre-OST0001 complete [ 1419.786407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1422.704323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1425.415922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1428.468065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1434.440815] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1438.009449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.016900] Lustre: lustre-MDT0000: reset Object Index mappings [ 1438.163451] LustreError: 114227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.188715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.522290] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1442.207797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.215139] Lustre: lustre-MDT0001: reset Object Index mappings [ 1443.738519] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1444.620358] Lustre: 115375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1446.680830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1447.970616] LustreError: 115728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.973394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 1448.785437] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1451.563799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1452.843519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 1452.843816] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1452.844348] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 1453.665744] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1456.715114] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1457.854668] Lustre: 117248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1471.954490] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:49:56 (1784123396) [ 1503.712694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1503.714671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1503.718090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1503.719493] Lustre: Skipped 3 previous similar messages [ 1507.166843] Lustre: server umount lustre-MDT0000 complete [ 1508.322323] LustreError: 117250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123433 with bad export cookie 5601932111449277698 [ 1508.325341] LustreError: 117250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1508.327255] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1509.344950] LustreError: 117272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.349258] LustreError: 117272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1514.552298] Lustre: server umount lustre-MDT0001 complete [ 1530.336176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1530.374406] Lustre: server umount lustre-OST0000 complete [ 1537.749065] Lustre: server umount lustre-OST0001 complete [ 1539.695517] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 1542.431130] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1553.619362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1553.700403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1553.714715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1553.754336] Lustre: lustre-MDT0000: new disk, initializing [ 1553.780105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1555.101382] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1558.045089] Lustre: server umount lustre-MDT0000 complete [ 1561.093433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.186918] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1561.191984] Lustre: 121504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1561.196699] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1562.518035] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1564.130371] Lustre: server umount lustre-MDT0000 complete [ 1566.730907] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:51:31 (1784123491) [ 1567.200030] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1567.730141] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:51:32 (1784123492) [ 1572.487536] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1575.667849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.789225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1575.791179] Lustre: Skipped 5 previous similar messages [ 1577.064761] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1579.744301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.784440] Lustre: 123552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1579.806234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1579.807905] Lustre: Skipped 1 previous similar message [ 1579.841131] Lustre: lustre-MDT0001: new disk, initializing [ 1579.886164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1579.890683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1581.168645] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1583.339106] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1585.397928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1585.496482] Lustre: lustre-OST0000: new disk, initializing [ 1585.498893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1585.502307] Lustre: 125127:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1586.615305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1586.618439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1586.648147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1587.353858] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1591.362326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1591.425433] Lustre: 125998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1592.636658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1592.640078] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1592.666463] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1593.289385] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1597.519907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1598.592884] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1733.992133] Lustre: 132374:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1733.994777] Lustre: 132374:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1733.996750] Lustre: 132374:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1733.998684] Lustre: 132374:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1734.000811] Lustre: 132374:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1734.002844] Lustre: 132374:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1799.118905] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:55:23 (1784123723) [ 1810.914625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.915849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1810.919350] Lustre: Skipped 6 previous similar messages [ 1810.924593] Lustre: Skipped 10 previous similar messages [ 1815.520971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1816.772954] Lustre: server umount lustre-MDT0000 complete [ 1818.030035] LustreError: 123545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123742 with bad export cookie 5601932111449330394 [ 1818.033597] LustreError: 123545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1818.034296] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1821.153451] LustreError: 126525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1821.157981] LustreError: 126525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1824.244845] Lustre: server umount lustre-MDT0001 complete [ 1831.492276] Lustre: server umount lustre-OST0000 complete [ 1847.264142] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1847.334128] Lustre: server umount lustre-OST0001 complete [ 1849.535117] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 1852.783017] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1864.737971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.823832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1864.833871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1864.836520] Lustre: Skipped 1 previous similar message [ 1864.869934] Lustre: lustre-MDT0000: new disk, initializing [ 1864.871259] Lustre: Skipped 1 previous similar message [ 1864.892368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.894546] Lustre: Skipped 3 previous similar messages [ 1864.899841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1866.348977] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1870.634790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.673345] Lustre: 135384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1870.766273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1872.003861] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1875.575883] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1877.283205] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1879.339562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1879.434714] Lustre: 137351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1881.086541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1881.089331] Lustre: Skipped 1 previous similar message [ 1881.091599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1881.134787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1881.293693] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 1884.773540] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1887.567263] Lustre: lustre-MDD0000: changelog on [ 1888.409911] Lustre: lustre-MDD0001: changelog on [ 2756.214124] Lustre: lustre-MDD0001: changelog off [ 2757.215976] Lustre: 135390:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2757.672461] Lustre: lustre-MDD0000: changelog off [ 2761.698056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2761.703538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.703936] LustreError: Skipped 1 previous similar message [ 2761.706778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2761.718813] Lustre: Skipped 5 previous similar messages [ 2761.720950] Lustre: Skipped 9 previous similar messages [ 2766.342043] Lustre: server umount lustre-OST0000 complete [ 2771.942346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.942615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2774.592394] Lustre: server umount lustre-MDT0000 complete [ 2776.774605] LustreError: 135375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124701 with bad export cookie 5601932111450092029 [ 2776.776244] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2776.785030] LustreError: 135375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2776.958808] Lustre: server umount lustre-MDT0001 complete [ 2782.597201] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 2784.148507] Key type lgssc unregistered [ 2784.328225] LNet: 146275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2784.334977] LNetError: 146275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2785.387336] LNet: Removed LNI 192.168.201.131@tcp [ 2785.941702] Key type .llcrypt unregistered [ 2785.943500] Key type ._llcrypt unregistered [ 2796.263055] Key type ._llcrypt registered [ 2796.265315] Key type .llcrypt registered [ 2796.332198] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 2803.962207] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2804.293369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2804.318067] alg: No test for adler32 (adler32-zlib) [ 2805.190576] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 2805.291708] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 2806.880591] Key type lgssc registered [ 2807.540521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2840.695269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2840.708343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.873656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2841.892253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2841.955338] Lustre: lustre-MDT0000: new disk, initializing [ 2842.011509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2842.026445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2846.227458] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 2857.691442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.823678] Lustre: 149456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2857.893297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2857.901270] Lustre: Skipped 1 previous similar message [ 2857.982250] Lustre: lustre-MDT0001: new disk, initializing [ 2858.125112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2858.178145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2858.198897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2864.190416] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 2873.048514] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2877.501655] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2881.823356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2882.043416] Lustre: lustre-OST0000: new disk, initializing [ 2882.045954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2882.050377] Lustre: 151419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2882.132987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2886.178616] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 2888.244923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2888.262697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2888.334770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2892.362548] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2896.900941] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_module obdecho/obdecho [ 2898.210988] LustreError: 152423:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2898.216365] LustreError: 152423:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2899.348413] LustreError: 152520:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 2909.153308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.157628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2909.163248] Lustre: Skipped 2 previous similar messages [ 2909.168210] Lustre: Skipped 2 previous similar messages [ 2911.301642] Lustre: server umount lustre-MDT0000 complete [ 2913.117691] LustreError: 149447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124837 with bad export cookie 218377706733028197 [ 2913.122451] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2913.124561] LustreError: 149447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2913.271984] Lustre: server umount lustre-MDT0001 complete [ 2924.459856] Lustre: server umount lustre-OST0000 complete [ 2936.316339] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 2942.636717] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2966.968950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2967.218439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2967.240429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2967.307727] Lustre: lustre-MDT0000: new disk, initializing [ 2967.374835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.396355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2970.630708] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 2978.023848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2978.115407] Lustre: 156232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2978.151667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2978.154289] Lustre: Skipped 1 previous similar message [ 2978.205349] Lustre: lustre-MDT0001: new disk, initializing [ 2978.258346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2978.302268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2978.315600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2981.034534] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 2987.635339] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2992.256138] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2997.544980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.779394] Lustre: lustre-OST0000: new disk, initializing [ 2997.782442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2997.786821] Lustre: 158198:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2997.854373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3002.999550] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3005.513175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3005.522181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3005.611630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3009.583336] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3012.567870] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3012.683552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3015.644945] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3015.803197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3021.290074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3021.290444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.295341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.299537] Lustre: Skipped 1 previous similar message [ 3023.232860] Lustre: server umount lustre-OST0000 complete [ 3026.405836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.411362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3026.415791] Lustre: Skipped 1 previous similar message [ 3026.425429] Lustre: Skipped 2 previous similar messages [ 3031.521835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.524702] Lustre: Skipped 1 previous similar message [ 3032.356171] Lustre: server umount lustre-MDT0000 complete [ 3035.031310] LustreError: 157034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124959 with bad export cookie 218377706733029156 [ 3035.036677] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3035.040076] LustreError: 157034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3035.242148] Lustre: server umount lustre-MDT0001 complete [ 3037.581828] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:16:01 (1784124961) [ 3038.390242] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3039.506371] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:16:03 (1784124963) [ 3044.098622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.363846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.653990] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3049.441245] LustreError: 160609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.449053] LustreError: 160609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3051.175840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.953832] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3058.610508] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3062.166629] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3066.505139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.699310] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3075.258821] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3085.881053] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.31@tcp (stopping) [ 3087.019767] Lustre: server umount lustre-MDT0000 complete [ 3087.328809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3087.335591] LustreError: Skipped 1 previous similar message [ 3087.341221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.346954] Lustre: Skipped 2 previous similar messages [ 3087.348566] LustreError: 160605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3087.355037] LustreError: 160605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3089.129395] LustreError: 160589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125013 with bad export cookie 218377706733030038 [ 3089.130762] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3089.136307] LustreError: 160589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3090.979570] LustreError: 160603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.452131] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3094.497103] LustreError: 161636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.498209] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3094.504730] LustreError: 161636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3094.515283] Lustre: Skipped 2 previous similar messages [ 3095.395826] Lustre: server umount lustre-MDT0001 complete [ 3099.609220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3106.768037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.907755] LustreError: 164446:0:(llog.c:1655:llog_backup()) MGC192.168.201.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3106.914060] Lustre: 164446:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3111.276478] LustreError: 164451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.717746] LustreError: 164452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.737070] LustreError: 164452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3138.530183] LustreError: 164451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3138.545544] LustreError: 164451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3168.224292] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3168.235885] Lustre: 164446:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.131@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3171.876354] LustreError: 164452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.886852] LustreError: 164452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 3208.730808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3208.736990] Lustre: Skipped 2 previous similar messages [ 3211.262754] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3213.795110] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3216.517985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.119707] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3238.385888] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3240.351494] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3241.277609] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3245.014100] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3246.165111] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3256.965705] Lustre: server umount lustre-MDT0000 complete [ 3258.806587] LustreError: 160590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125183 with bad export cookie 218377706733036485 [ 3258.812935] LustreError: 160590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3258.813534] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3258.849273] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.855429] Lustre: Skipped 1 previous similar message [ 3258.857405] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3258.860915] LustreError: 164453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.869052] LustreError: 164453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 3259.013532] Lustre: server umount lustre-MDT0001 complete [ 3263.097615] Lustre: server umount lustre-OST0000 complete [ 3275.441692] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 3280.571230] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3299.502898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.630804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3299.645284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3299.707477] Lustre: lustre-MDT0000: new disk, initializing [ 3299.750656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3299.753932] Lustre: Skipped 1 previous similar message [ 3299.761650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3301.653788] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3307.658055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.708215] Lustre: 169520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3307.859145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3309.922950] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3314.930807] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3318.209393] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3321.831870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3321.994579] Lustre: 171485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3324.795323] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3329.658600] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3330.565633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3330.618934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3331.846686] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3331.963559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3334.107423] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3334.229890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3335.650049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3335.651141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3335.653323] LustreError: Skipped 1 previous similar message [ 3335.662843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3337.241436] Lustre: server umount lustre-OST0000 complete [ 3345.024344] Lustre: server umount lustre-MDT0000 complete [ 3346.798801] LustreError: 169512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125271 with bad export cookie 218377706733053313 [ 3346.799112] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3346.804267] LustreError: 169512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3346.975514] Lustre: server umount lustre-MDT0001 complete [ 3351.330204] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:21:15 (1784125275) [ 3354.395820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.476569] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3360.162674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.127817] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3365.391643] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3367.576963] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3370.362790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.629497] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3375.634937] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3386.048602] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:21:50 (1784125310) [ 3389.922176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3389.923087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.928832] Lustre: Skipped 4 previous similar messages [ 3393.568184] Lustre: server umount lustre-OST0000 complete [ 3396.066056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3400.161403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3400.164023] Lustre: Skipped 7 previous similar messages [ 3401.317104] Lustre: server umount lustre-MDT0000 complete [ 3402.848118] LustreError: 173874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125327 with bad export cookie 218377706733054181 [ 3402.851800] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3402.859079] LustreError: 173874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3403.011757] Lustre: server umount lustre-MDT0001 complete [ 3407.050607] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 3408.138730] Key type lgssc unregistered [ 3408.269328] LNet: 177628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3408.271569] LNetError: 177628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3408.279264] LNet: Removed LNI 192.168.201.131@tcp [ 3408.600671] Key type .llcrypt unregistered [ 3408.602269] Key type ._llcrypt unregistered [ 3417.225154] Key type ._llcrypt registered [ 3417.226168] Key type .llcrypt registered [ 3417.270371] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3417.630224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3417.641305] alg: No test for adler32 (adler32-zlib) [ 3418.506206] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 3418.603175] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 3420.192240] Key type lgssc registered [ 3420.717492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3424.499912] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3424.509276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3439.968987] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3445.088304] LustreError: 178849:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.131@tcp: failed processing log, type 4: rc = -110 [ 3450.208448] LustreError: 178849:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3450.215304] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3450.218385] LustreError: MGC192.168.201.131@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3450.223440] LustreError: 178849:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3450.227618] LustreError: 178849:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 3450.231273] LustreError: 178849:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3450.259399] Lustre: server umount lustre-OST0000 complete [ 3450.261803] LustreError: 178849:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3454.420819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.692697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3457.800422] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3461.257880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.414383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3463.015464] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3466.165596] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3468.664774] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3471.405050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3471.482966] LustreError: 181160:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3471.487966] Lustre: 181160:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.131@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3471.539042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.770975] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3476.665784] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3477.987093] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3477.988296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3477.991639] Lustre: Skipped 2 previous similar messages [ 3477.996160] Lustre: Skipped 2 previous similar messages [ 3483.106402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3483.109856] Lustre: Skipped 2 previous similar messages [ 3483.551273] Lustre: server umount lustre-MDT0000 complete [ 3485.257229] LustreError: 179329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125410 with bad export cookie 3453010151433802657 [ 3485.261254] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3485.262111] LustreError: 179329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3485.416383] Lustre: server umount lustre-MDT0001 complete [ 3497.503689] Lustre: server umount lustre-OST0000 complete [ 3501.186128] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 3505.489146] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3521.318431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3521.430437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3521.444640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3521.491949] Lustre: lustre-MDT0000: new disk, initializing [ 3521.530435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3521.537519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3523.305187] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3527.944861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3527.986749] Lustre: 184678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3528.009288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3528.011199] Lustre: Skipped 1 previous similar message [ 3528.046635] Lustre: lustre-MDT0001: new disk, initializing [ 3528.079587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3528.093919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3528.098515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3529.460375] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3533.567466] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3535.759426] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3538.504167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3538.613870] Lustre: lustre-OST0000: new disk, initializing [ 3538.616158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3538.619361] Lustre: 186646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3538.650510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3540.951271] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3544.805238] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3546.104206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3546.107429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3546.143455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3546.291126] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3546.360253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3547.857115] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3547.983345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3551.201446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3551.202773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.204454] LustreError: Skipped 1 previous similar message [ 3551.205089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.210029] Lustre: Skipped 1 previous similar message [ 3554.774879] Lustre: server umount lustre-OST0000 complete [ 3556.321599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3556.322515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3556.325725] Lustre: Skipped 1 previous similar message [ 3556.327517] Lustre: Skipped 1 previous similar message [ 3561.441580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3561.446916] Lustre: Skipped 2 previous similar messages [ 3562.372954] Lustre: server umount lustre-MDT0000 complete [ 3563.889141] LustreError: 184672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125488 with bad export cookie 3453010151433803343 [ 3563.892150] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.893740] LustreError: 184672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3564.017572] Lustre: server umount lustre-MDT0001 complete [ 3567.504092] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:24:51 (1784125491) [ 3574.824860] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 3576.059913] Key type lgssc unregistered [ 3576.205577] LNet: 189588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3576.208023] LNetError: 189588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3576.214338] LNet: Removed LNI 192.168.201.131@tcp [ 3576.592503] Key type .llcrypt unregistered [ 3576.594221] Key type ._llcrypt unregistered [ 3584.547824] Key type ._llcrypt registered [ 3584.549379] Key type .llcrypt registered [ 3584.592241] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3584.911924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3584.927513] alg: No test for adler32 (adler32-zlib) [ 3585.788884] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 3585.893993] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 3587.480160] Key type lgssc registered [ 3587.883311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3590.912594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3590.919789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.083663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.583967] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3597.114901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.292505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3599.048443] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3602.175991] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3604.507045] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3607.696324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3607.799378] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3607.801124] LustreError: 192644:0:(llog.c:1696:llog_backup()) MGC192.168.201.131@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3607.805046] Lustre: 192644:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.131@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3607.858387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.258713] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3613.192677] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3617.760836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3617.764401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3617.769073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.273133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.274299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3618.276430] Lustre: Skipped 1 previous similar message [ 3618.281368] Lustre: Skipped 1 previous similar message [ 3620.252433] Lustre: server umount lustre-MDT0000 complete [ 3621.687917] LustreError: 190763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125546 with bad export cookie 4274646384937892731 [ 3621.689330] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3621.691994] LustreError: 190763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3621.811238] Lustre: server umount lustre-MDT0001 complete [ 3633.557372] Lustre: server umount lustre-OST0000 complete [ 3636.968723] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 3640.306195] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3653.688106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3653.768388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3653.779663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3653.816335] Lustre: lustre-MDT0000: new disk, initializing [ 3653.843150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.850740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3655.343993] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3660.590882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.640837] Lustre: 196161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3660.659876] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3660.661833] Lustre: Skipped 1 previous similar message [ 3660.663470] LustreError: 196940:0:(llog.c:1696:llog_backup()) MGC192.168.201.131@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3660.666908] LustreError: 196940:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 3660.670740] Lustre: 196940:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.131@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3660.681466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3660.684699] Lustre: Skipped 1 previous similar message [ 3660.730102] Lustre: lustre-MDT0001: new disk, initializing [ 3660.772459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3660.794316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3660.799357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3662.450853] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3666.540573] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3668.518227] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3671.005116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3671.108923] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3671.110609] Lustre: Skipped 1 previous similar message [ 3671.112281] LustreError: 198102:0:(llog.c:1696:llog_backup()) MGC192.168.201.131@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3671.116237] LustreError: 198102:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 3671.119861] Lustre: 198102:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.131@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3671.123640] Lustre: 198102:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3671.133166] Lustre: lustre-OST0000: new disk, initializing [ 3671.135268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3671.138090] Lustre: 198125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3671.174303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3673.323444] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3677.320665] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3679.013570] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3680.253867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3680.258101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3680.286177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3681.138201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3682.671995] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3682.749553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3685.345591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3685.346710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.348310] LustreError: Skipped 1 previous similar message [ 3685.352896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3685.358422] Lustre: Skipped 1 previous similar message [ 3689.558733] Lustre: server umount lustre-OST0000 complete [ 3691.488858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3691.491774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3691.495855] Lustre: Skipped 1 previous similar message [ 3691.497652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3695.585375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3695.585811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.312252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3705.382808] Lustre: server umount lustre-MDT0000 complete [ 3705.825299] LustreError: 196968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3705.837020] LustreError: 196968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3707.078791] LustreError: 196952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125631 with bad export cookie 4274646384937893410 [ 3707.079426] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3707.087671] LustreError: 196952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3707.238458] Lustre: server umount lustre-MDT0001 complete [ 3710.928777] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:27:15 (1784125635) [ 3717.388264] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3721.297641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.434821] LustreError: 201263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.450173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.919533] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3726.447553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.262205] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3729.374626] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3731.628655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3734.133841] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3737.785718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3737.859580] Lustre: lustre-OST0001: new disk, initializing [ 3737.861479] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3737.864304] Lustre: 203568:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3740.231620] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3746.208232] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3746.830845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3746.834882] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3746.876247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3749.241545] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3750.126159] Lustre: DEBUG MARKER: ADD OST3 [ 3752.255663] loop: module loaded [ 3752.276432] loop0: detected capacity change from 0 to 204800000 [ 3752.361142] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3752.365904] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3752.370556] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3754.908357] loop0: detected capacity change from 0 to 204800000 [ 3756.978678] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3757.058319] Lustre: lustre-OST0003: new disk, initializing [ 3757.060300] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3757.062685] Lustre: 205299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3757.096801] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3757.100151] Lustre: Skipped 3 previous similar messages [ 3757.207744] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3757.260508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3757.264790] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3757.277586] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3757.500076] Lustre: DEBUG MARKER: STOP OST3 [ 3758.267134] Lustre: Failing over lustre-OST0003 [ 3758.316911] Lustre: server umount lustre-OST0003 complete [ 3758.961113] LustreError: 202750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3759.264053] LustreError: 205331:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3759.468283] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3762.662895] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3762.664796] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.666330] LustreError: 203439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.670853] Lustre: Skipped 1 previous similar message [ 3762.962780] Lustre: DEBUG MARKER: START OST3 again [ 3765.576840] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3765.703722] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3767.396757] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3767.408964] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3767.409042] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3768.078283] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3782.155780] Lustre: Failing over lustre-OST0003 [ 3782.191459] Lustre: server umount lustre-OST0003 complete [ 3783.137639] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3783.140740] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3783.146511] LustreError: 202748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3783.154201] LustreError: 202748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3791.332932] LustreError: 202749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.338325] LustreError: 202749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3796.457140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3796.459142] Lustre: Skipped 6 previous similar messages [ 3809.250622] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3809.331437] Lustre: server umount lustre-MDT0000 complete [ 3810.775134] LustreError: 202397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125735 with bad export cookie 4274646384937894278 [ 3810.779069] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3810.780115] LustreError: 202397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3811.809947] LustreError: 203438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.818786] LustreError: 203438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3815.841936] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3815.846012] Lustre: Skipped 11 previous similar messages [ 3825.120076] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3825.244070] Lustre: server umount lustre-MDT0001 complete [ 3826.985525] Lustre: server umount lustre-OST0000 complete [ 3828.772760] Lustre: server umount lustre-OST0001 complete [ 3831.589540] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 3834.573286] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3849.701598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.796337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3849.811215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3849.852159] Lustre: lustre-MDT0000: new disk, initializing [ 3849.881746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3849.884211] Lustre: Skipped 1 previous similar message [ 3849.890054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3851.373316] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3855.813659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.857334] Lustre: 210714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3855.970674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3857.401188] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3861.642710] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3863.846263] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3866.588396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3866.700635] Lustre: lustre-OST0000: new disk, initializing [ 3866.702088] Lustre: Skipped 1 previous similar message [ 3866.704607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3866.708249] Lustre: Skipped 2 previous similar messages [ 3866.711243] Lustre: 212680:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3867.967956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3867.974214] Lustre: Skipped 1 previous similar message [ 3867.977681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3868.016548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3868.884181] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3873.138496] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3874.966577] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3875.061406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3876.665334] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3876.737052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3878.372209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3878.373451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3878.374920] LustreError: Skipped 2 previous similar messages [ 3878.375481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3878.378962] Lustre: Skipped 8 previous similar messages [ 3878.384738] Lustre: Skipped 3 previous similar messages [ 3883.475414] Lustre: server umount lustre-OST0000 complete [ 3899.360164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3899.442316] Lustre: server umount lustre-MDT0000 complete [ 3900.855229] LustreError: 210706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125825 with bad export cookie 4274646384937896553 [ 3900.858850] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3900.863029] LustreError: 210706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3900.981204] Lustre: server umount lustre-MDT0001 complete [ 3902.392404] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:30:26 (1784125826) [ 3909.755766] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 3913.973069] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3927.948345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.046635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3928.058604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3928.098448] Lustre: lustre-MDT0000: new disk, initializing [ 3928.135099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3929.782442] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3934.415538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.462261] Lustre: 217285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3934.577447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3936.211867] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3940.479453] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3942.743418] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3945.318938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3945.439946] Lustre: 219254:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3947.397808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3947.797623] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3952.113844] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3953.889722] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3953.981737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3955.695685] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3955.788636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3957.731244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.732743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3957.737312] Lustre: Skipped 2 previous similar messages [ 3957.741376] LustreError: Skipped 2 previous similar messages [ 3957.742402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3957.747197] Lustre: Skipped 8 previous similar messages [ 3962.717437] Lustre: server umount lustre-OST0000 complete [ 3970.668114] Lustre: server umount lustre-MDT0000 complete [ 3972.241270] LustreError: 217319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125897 with bad export cookie 4274646384937897435 [ 3972.244185] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3972.245763] LustreError: 217319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3972.375512] Lustre: server umount lustre-MDT0001 complete [ 3975.579697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.752549] LustreError: 221467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.759976] LustreError: 221467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3977.276752] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3980.900451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3981.099166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3981.103064] Lustre: Skipped 6 previous similar messages [ 3982.801126] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3986.450858] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3988.887158] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3991.611174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3994.022913] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 3996.892408] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4011.493989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4011.497170] LustreError: Skipped 1 previous similar message [ 4015.123754] Lustre: server umount lustre-OST0000 complete [ 4022.619734] Lustre: server umount lustre-MDT0000 complete [ 4023.920171] LustreError: 224064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125948 with bad export cookie 4274646384937898317 [ 4023.922911] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.923489] LustreError: 224064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4024.044535] Lustre: server umount lustre-MDT0001 complete [ 4129.487477] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4130.529328] Key type lgssc unregistered [ 4130.666178] LNet: 225150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4130.669434] LNetError: 225150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4130.679334] LNet: Removed LNI 192.168.201.131@tcp [ 4131.001149] Key type .llcrypt unregistered [ 4131.002540] Key type ._llcrypt unregistered [ 4138.910052] Key type ._llcrypt registered [ 4138.911140] Key type .llcrypt registered [ 4138.944213] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 4145.273658] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4145.681548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4145.688765] alg: No test for adler32 (adler32-zlib) [ 4146.548421] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 4146.638101] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 4148.232153] Key type lgssc registered [ 4148.607480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4160.388994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4160.395774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.478665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4161.489782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4161.524045] Lustre: lustre-MDT0000: new disk, initializing [ 4161.553228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4161.561150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4163.033591] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4167.271691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.318071] Lustre: 228474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4167.346244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4167.348452] Lustre: Skipped 1 previous similar message [ 4167.385497] Lustre: lustre-MDT0001: new disk, initializing [ 4167.416608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.428370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4167.431530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4168.935709] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4173.128789] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4175.361838] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4177.976071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4178.103315] Lustre: lustre-OST0000: new disk, initializing [ 4178.105922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4178.109838] Lustre: 230440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4178.153933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4178.817644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4178.835382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4178.865214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4180.513028] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4184.366423] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4186.142094] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4186.263977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4187.955642] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4188.034830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4189.154903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4189.157693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.158071] LustreError: Skipped 1 previous similar message [ 4189.163856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4189.167728] Lustre: Skipped 1 previous similar message [ 4194.274678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.276731] Lustre: Skipped 1 previous similar message [ 4194.777525] Lustre: server umount lustre-OST0000 complete [ 4198.368951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4198.371634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.377112] Lustre: Skipped 1 previous similar message [ 4198.379353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4199.394211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4202.136748] Lustre: server umount lustre-MDT0000 complete [ 4203.437245] LustreError: 228465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126128 with bad export cookie 17899481348452425029 [ 4203.440447] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4203.441093] LustreError: 228465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4203.563500] Lustre: server umount lustre-MDT0001 complete [ 4209.676245] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4213.030939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.185322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.528764] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4217.352291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4217.474776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4218.839799] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4219.845987] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4222.065178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4224.256891] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4227.325636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4227.394051] Lustre: lustre-OST0001: new disk, initializing [ 4227.396514] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4227.399826] Lustre: 235677:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4227.429813] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4227.431673] Lustre: Skipped 1 previous similar message [ 4228.744484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4228.748874] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4228.786676] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4229.510484] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4236.230821] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4240.140502] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4244.566458] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:36:08 (1784126168) [ 4248.545029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4248.547515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4248.551704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4248.554244] Lustre: Skipped 2 previous similar messages [ 4251.425060] Lustre: server umount lustre-MDT0000 complete [ 4252.729601] LustreError: 233350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126177 with bad export cookie 17899481348452425897 [ 4252.733513] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4252.733896] LustreError: 233350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4252.841166] Lustre: server umount lustre-MDT0001 complete [ 4264.902051] Lustre: server umount lustre-OST0000 complete [ 4275.355417] Lustre: server umount lustre-OST0001 complete [ 4277.425640] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 4280.736744] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4292.625781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.719329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4292.729935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4292.770461] Lustre: lustre-MDT0000: new disk, initializing [ 4292.791276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4292.797941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4294.189764] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4298.284164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4298.325712] Lustre: 239616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4298.423758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4299.766142] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4303.315246] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4305.171549] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4307.498461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4307.589917] Lustre: lustre-OST0000: new disk, initializing [ 4307.591233] Lustre: Skipped 1 previous similar message [ 4307.592572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4307.594293] Lustre: Skipped 2 previous similar messages [ 4307.596056] Lustre: 241586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4309.181087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4309.184768] Lustre: Skipped 1 previous similar message [ 4309.187369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4309.225931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4309.610363] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4313.116760] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4314.457767] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4314.530642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4315.868819] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4315.936505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4319.714446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4319.714699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4319.719000] Lustre: Skipped 3 previous similar messages [ 4319.721220] LustreError: Skipped 1 previous similar message [ 4319.723738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4319.726515] Lustre: Skipped 5 previous similar messages [ 4322.640389] Lustre: server umount lustre-OST0000 complete [ 4324.321020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4329.953261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4329.955420] Lustre: Skipped 3 previous similar messages [ 4338.144101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4338.207545] Lustre: server umount lustre-MDT0000 complete [ 4339.451166] LustreError: 241585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126264 with bad export cookie 17899481348452427990 [ 4339.454209] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4339.456232] LustreError: 241585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4339.565655] Lustre: server umount lustre-MDT0001 complete [ 4342.195640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4342.362812] LustreError: 243800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4342.368408] LustreError: 243800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4342.387340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4342.389501] Lustre: Skipped 2 previous similar messages [ 4343.672409] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4346.306786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4347.806561] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4350.190531] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4352.020161] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4354.430699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4356.687340] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4359.300969] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4504.034722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4504.035796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4504.039048] Lustre: Skipped 4 previous similar messages [ 4504.040545] Lustre: Skipped 2 previous similar messages [ 4518.368115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4518.421592] Lustre: server umount lustre-OST0000 complete [ 4520.416881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4520.420483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4525.714062] Lustre: server umount lustre-MDT0000 complete [ 4526.932298] LustreError: 244534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126451 with bad export cookie 17899481348452428872 [ 4526.934511] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4526.937780] LustreError: 244534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4527.045081] Lustre: server umount lustre-MDT0001 complete [ 4624.927760] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4625.912777] Key type lgssc unregistered [ 4626.037290] LNet: 247408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4626.039392] LNetError: 247408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4626.048332] LNet: Removed LNI 192.168.201.131@tcp [ 4626.340114] Key type .llcrypt unregistered [ 4626.341357] Key type ._llcrypt unregistered [ 4634.571778] Key type ._llcrypt registered [ 4634.573187] Key type .llcrypt registered [ 4634.615711] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 4640.691915] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4641.132190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4641.142303] alg: No test for adler32 (adler32-zlib) [ 4641.999151] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 4642.094649] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 4643.680168] Key type lgssc registered [ 4644.038235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4658.883648] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4662.476711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4662.483875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4663.567530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4663.579322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4663.617517] Lustre: lustre-MDT0000: new disk, initializing [ 4663.644493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.651149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4665.044293] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4669.180759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4669.220521] Lustre: 251449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4669.243125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4669.245132] Lustre: Skipped 1 previous similar message [ 4669.282272] Lustre: lustre-MDT0001: new disk, initializing [ 4669.316698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4669.329482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4669.332813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4670.677051] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4673.167090] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4675.721709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4675.840763] Lustre: lustre-OST0000: new disk, initializing [ 4675.842893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4675.846071] Lustre: 253085:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4675.892190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.980833] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4682.290460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4682.350467] Lustre: lustre-OST0001: new disk, initializing [ 4682.352599] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4682.356612] Lustre: 253943:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4682.392441] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4683.221620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4683.227550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4683.310406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4683.310413] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4684.591072] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4690.113199] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4698.016821] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4703.718205] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4703.720234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4703.723132] Lustre: Skipped 3 previous similar messages [ 4703.725966] Lustre: Skipped 2 previous similar messages [ 4707.300526] Lustre: server umount lustre-MDT0000 complete [ 4708.595207] LustreError: 251440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126633 with bad export cookie 1835812691734346667 [ 4708.598407] LustreError: 251440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4708.598845] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.730805] Lustre: server umount lustre-MDT0001 complete [ 4719.304248] Lustre: server umount lustre-OST0000 complete [ 4730.903483] Lustre: server umount lustre-OST0001 complete [ 4735.097186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4737.943510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4740.623685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4743.536015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4749.344205] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4752.762516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4752.770949] Lustre: lustre-MDT0000: reset Object Index mappings [ 4752.915781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4754.208512] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4757.169562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4757.175482] Lustre: lustre-MDT0001: reset Object Index mappings [ 4758.637076] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4759.641364] Lustre: 260088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4761.848851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4761.973633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4761.975610] Lustre: Skipped 1 previous similar message [ 4762.981263] LustreError: 260631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.983807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4762.989162] LustreError: 260631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4763.847890] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4766.719920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4767.851460] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4768.736590] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4771.941331] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4773.163789] Lustre: 261955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4777.952792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4777.954859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.958321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4780.442316] Lustre: server umount lustre-MDT0000 complete [ 4781.617095] LustreError: 260091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126706 with bad export cookie 1835812691734353359 [ 4781.619821] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4781.620701] LustreError: 260091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4781.730895] Lustre: server umount lustre-MDT0001 complete [ 4792.769736] Lustre: server umount lustre-OST0000 complete [ 4804.250555] Lustre: server umount lustre-OST0001 complete [ 4810.606606] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:45:35 (1784126735) [ 4815.589887] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 4818.436018] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4829.675144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.751084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4829.761757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4829.797858] Lustre: lustre-MDT0000: new disk, initializing [ 4829.817604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.819379] Lustre: Skipped 1 previous similar message [ 4829.826107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4829.828528] Lustre: Skipped 1 previous similar message [ 4831.160117] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4835.237023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.277765] Lustre: 266200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4835.383965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4835.387461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4835.391151] Lustre: Skipped 1 previous similar message [ 4836.639492] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4840.139814] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4841.889737] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4844.025734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4844.121317] Lustre: lustre-OST0000: new disk, initializing [ 4844.122933] Lustre: Skipped 1 previous similar message [ 4844.124313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4844.126416] Lustre: Skipped 2 previous similar messages [ 4844.128966] Lustre: 268168:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4845.951543] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4846.071289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4846.074294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4846.104975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4849.546474] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4850.931195] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4851.012085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4852.372262] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4852.436767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4855.124459] Lustre: server umount lustre-OST0000 complete [ 4860.896844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4860.899587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4860.905747] Lustre: Skipped 3 previous similar messages [ 4860.908216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4860.910711] Lustre: Skipped 4 previous similar messages [ 4866.529311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4866.531284] Lustre: Skipped 3 previous similar messages [ 4866.557316] Lustre: server umount lustre-MDT0000 complete [ 4867.709021] LustreError: 266191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126792 with bad export cookie 1835812691734355431 [ 4867.712284] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4867.713234] LustreError: 266191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4877.330215] Lustre: server umount lustre-MDT0001 complete [ 4882.756981] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4885.947963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.069436] LustreError: 271094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4886.082133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4886.084515] Lustre: Skipped 2 previous similar messages [ 4887.360269] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4890.230058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4891.672279] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4892.652838] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4894.783361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4896.735837] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4899.413252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4899.472957] Lustre: lustre-OST0001: new disk, initializing [ 4899.474363] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4899.476410] Lustre: 273400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4900.736877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4900.741801] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4900.769546] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4901.381154] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4905.684476] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4906.807939] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4908.371385] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4908.378441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4911.076251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.077358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4911.077830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4911.080548] Lustre: Skipped 3 previous similar messages [ 4916.193403] Lustre: server umount lustre-MDT0000 complete [ 4916.194126] LustreError: 271094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4916.198519] LustreError: 271094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4917.416170] LustreError: 271077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126842 with bad export cookie 1835812691734356313 [ 4917.419465] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4917.420629] LustreError: 271077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4917.534775] Lustre: server umount lustre-MDT0001 complete [ 4928.517818] Lustre: server umount lustre-OST0000 complete [ 4939.876263] Lustre: server umount lustre-OST0001 complete [ 4945.076611] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4948.292135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.402584] LustreError: 276458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4949.605852] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4952.207982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.329287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4952.331380] Lustre: Skipped 4 previous similar messages [ 4953.571916] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4954.451424] Lustre: 277600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4956.418379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4958.446090] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4958.568177] LustreError: 277954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4958.578348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4961.158994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4962.286316] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4963.134389] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 4966.252687] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4967.481936] Lustre: 279474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4969.197074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4969.205167] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4972.152442] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:48:16 (1784126896) [ 4987.881450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.887964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4987.890102] Lustre: Skipped 4 previous similar messages [ 4987.891850] Lustre: Skipped 5 previous similar messages [ 4989.727716] Lustre: server umount lustre-MDT0000 complete [ 4991.132265] LustreError: 276439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126915 with bad export cookie 1835812691734362627 [ 4991.135039] LustreError: 276439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4991.136118] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4992.993279] LustreError: 277191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4992.997932] LustreError: 277191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4997.307850] Lustre: server umount lustre-MDT0001 complete [ 5004.734418] Lustre: server umount lustre-OST0000 complete [ 5012.064448] Lustre: server umount lustre-OST0001 complete [ 5014.244381] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 5017.393774] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5032.604572] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5036.133416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5036.219343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5036.230060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5036.268852] Lustre: lustre-MDT0000: new disk, initializing [ 5036.297558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5037.680135] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5041.877154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5041.920238] Lustre: 283703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5042.036044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5043.491034] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5047.882202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5047.991091] Lustre: 285338:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5048.673782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5049.973729] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5054.257729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5055.730728] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5056.406714] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5060.903168] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5065.664506] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:49:50 (1784126990) [ 5067.744923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5067.748345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5067.752377] Lustre: Skipped 2 previous similar messages [ 5067.753855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5067.755918] Lustre: Skipped 5 previous similar messages [ 5072.473540] Lustre: server umount lustre-MDT0000 complete [ 5073.694165] LustreError: 283698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126998 with bad export cookie 1835812691734370495 [ 5073.700070] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5073.817841] Lustre: server umount lustre-MDT0001 complete [ 5084.678580] Lustre: server umount lustre-OST0000 complete [ 5096.225165] Lustre: server umount lustre-OST0001 complete [ 5099.159413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5099.322393] LustreError: 288449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5099.329070] LustreError: 288449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5099.356234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5099.358258] Lustre: Skipped 6 previous similar messages [ 5100.765449] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5103.577892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.112603] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5107.634387] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5109.542082] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5111.643657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5113.719410] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5115.945889] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5131.808443] LustreError: 290661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.812803] LustreError: 290661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5132.688913] loop1: detected capacity change from 0 to 204800000 [ 5132.758770] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5132.763117] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5132.766762] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5134.478145] loop1: detected capacity change from 0 to 204800000 [ 5136.217617] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5136.246633] LustreError: 288439:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs2674: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5136.250640] LustreError: 288439:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs2674 [ 5136.252341] LustreError: 288439:0:(mgs_handler.c:517:mgs_target_reg()) fs2674-MDT0001: Error checking target index: rc = -13 [ 5136.255456] LustreError: fs2674-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5136.258830] LustreError: 292183:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5136.261496] LustreError: 292183:0:(tgt_mount.c:2129:server_put_super()) no obd fs2674-MDT0001 [ 5136.263304] LustreError: 292183:0:(tgt_mount.c:116:server_deregister_mount()) fs2674-MDT0001 not registered [ 5136.273896] Lustre: server umount fs2674-MDT0001 complete [ 5136.275607] LustreError: 292183:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5139.149764] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5139.182144] Lustre: Setting parameter fs2674-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs2674-MDT0001 [ 5139.185785] Lustre: Skipped 2 previous similar messages [ 5139.202809] Lustre: srv-fs2674-MDT0001: No data found on store. Initialize space. [ 5139.205244] Lustre: Skipped 4 previous similar messages [ 5139.242607] Lustre: fs2674-MDT0001: new disk, initializing [ 5139.244448] Lustre: Skipped 3 previous similar messages [ 5139.278273] Lustre: srv-fs2674-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5139.281067] Lustre: 292648:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2674-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5139.285104] LustreError: 292648:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2674-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5140.528867] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5143.353345] Lustre: server umount fs2674-MDT0001 complete [ 5147.757916] loop1: detected capacity change from 0 to 204800000 [ 5149.492052] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5149.596066] Lustre: 294214:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2674-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5149.599438] LustreError: 294214:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2674-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5150.796840] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5152.252857] Lustre: server umount fs2674-MDT0001 complete [ 5156.834623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5156.836082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5156.838221] Lustre: Skipped 5 previous similar messages [ 5156.839851] Lustre: Skipped 5 previous similar messages [ 5161.308239] Lustre: server umount lustre-MDT0000 complete [ 5164.247633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.285839] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5165.758887] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5168.388704] loop1: detected capacity change from 0 to 204800000 [ 5169.641239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5170.233266] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5170.344792] Lustre: 296657:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2674-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5170.348219] LustreError: 296657:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2674-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5171.687492] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5173.318400] Lustre: server umount fs2674-MDT0001 complete [ 5180.385057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5186.386206] Lustre: server umount lustre-OST0000 complete [ 5190.113939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5190.114035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.117471] LustreError: Skipped 1 previous similar message [ 5190.122227] Lustre: Skipped 2 previous similar messages [ 5193.756500] Lustre: server umount lustre-MDT0000 complete [ 5195.054765] LustreError: 297959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784127119 with bad export cookie 1835812691734373169 [ 5195.058291] LustreError: 297959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5195.190265] Lustre: server umount lustre-MDT0001 complete [ 5199.086940] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5200.047290] Key type lgssc unregistered [ 5200.172353] LNet: 298998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5200.174957] LNetError: 298998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5200.189291] LNet: Removed LNI 192.168.201.131@tcp [ 5200.495106] Key type .llcrypt unregistered [ 5200.496162] Key type ._llcrypt unregistered [ 5207.962475] Key type ._llcrypt registered [ 5207.963412] Key type .llcrypt registered [ 5207.997844] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5208.420208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5208.431246] alg: No test for adler32 (adler32-zlib) [ 5209.284571] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 5209.369693] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 5210.944100] Key type lgssc registered [ 5211.281379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5213.698860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5213.705530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.845350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5216.161579] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5218.919403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5219.057224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5220.413306] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5222.808055] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5224.461047] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5226.747209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5226.870788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5228.767098] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5228.901396] LustreError: 302074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.908982] LustreError: 302074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5230.942688] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5234.146105] LustreError: 302075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5239.267431] LustreError: 302074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5241.382546] LustreError: 302076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5241.390816] LustreError: 302076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5245.412817] LustreError: 302459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5245.419468] LustreError: 302459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5247.598028] loop1: detected capacity change from 0 to 204800000 [ 5247.661641] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5247.664814] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5247.668532] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5249.373049] loop1: detected capacity change from 0 to 204800000 [ 5251.245783] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5251.299632] Lustre: fs19784-OST0000: new disk, initializing [ 5251.301202] Lustre: srv-fs19784-OST0000: No data found on store. Initialize space. [ 5251.303682] Lustre: 304000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5251.326288] Lustre: fs19784-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5253.279161] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5255.651403] LustreError: 303868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5255.656700] LustreError: 303868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5256.761460] Lustre: server umount fs19784-OST0000 complete [ 5260.981249] loop1: detected capacity change from 0 to 204800000 [ 5262.939684] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5263.013281] Lustre: fs19784-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5265.102309] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5267.796101] loop2: detected capacity change from 0 to 204800000 [ 5267.872673] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5267.876276] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5267.879438] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5269.489775] loop2: detected capacity change from 0 to 204800000 [ 5271.349828] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5271.385065] LustreError: fs19784-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 5271.390200] LustreError: fs19784-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5271.394117] LustreError: 307035:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5271.397669] LustreError: 307035:0:(tgt_mount.c:2129:server_put_super()) no obd fs19784-OST0001 [ 5271.399974] LustreError: 307035:0:(tgt_mount.c:116:server_deregister_mount()) fs19784-OST0001 not registered [ 5271.411235] Lustre: server umount fs19784-OST0001 complete [ 5271.412941] LustreError: 307035:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5272.096405] LustreError: 302076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5272.101386] LustreError: 302076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5274.712270] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5274.759056] Lustre: fs19784-OST0001: new disk, initializing [ 5274.761078] Lustre: srv-fs19784-OST0001: No data found on store. Initialize space. [ 5274.763666] Lustre: 307585:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5274.790033] Lustre: fs19784-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5276.642034] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5280.046714] Lustre: server umount fs19784-OST0000 complete [ 5283.131462] Lustre: server umount fs19784-OST0001 complete [ 5287.224708] loop1: detected capacity change from 0 to 204800000 [ 5289.106041] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5291.060144] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5293.265860] loop2: detected capacity change from 0 to 204800000 [ 5295.125678] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5295.185065] Lustre: fs19784-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5295.187601] Lustre: Skipped 1 previous similar message [ 5297.182611] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5300.289729] Lustre: server umount fs19784-OST0000 complete [ 5303.456038] Lustre: server umount fs19784-OST0001 complete [ 5306.852137] LustreError: 302459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.857292] LustreError: 302459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 5310.944992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5310.947688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5310.952585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5311.970064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.970069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5311.971580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5311.979757] Lustre: Skipped 1 previous similar message [ 5313.492390] Lustre: server umount lustre-OST0000 complete [ 5316.577035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5316.581197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5316.588657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5322.209135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5322.211221] Lustre: Skipped 3 previous similar messages [ 5327.329464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5327.331509] Lustre: Skipped 1 previous similar message [ 5329.376118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5329.439132] Lustre: server umount lustre-MDT0000 complete [ 5330.626212] LustreError: 300226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784127255 with bad export cookie 9062816080979928289 [ 5330.630134] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5330.631411] LustreError: 300226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5330.745414] Lustre: server umount lustre-MDT0001 complete [ 5334.304294] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5335.319266] Key type lgssc unregistered [ 5335.453333] LNet: 313497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5335.456847] LNetError: 313497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5335.466301] LNet: Removed LNI 192.168.201.131@tcp [ 5335.795107] Key type .llcrypt unregistered [ 5335.796220] Key type ._llcrypt unregistered [ 5346.198726] Key type ._llcrypt registered [ 5346.199830] Key type .llcrypt registered [ 5346.233845] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 5352.236507] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5352.572853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5352.631687] alg: No test for adler32 (adler32-zlib) [ 5353.484622] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 5353.573767] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 5355.152151] Key type lgssc registered [ 5355.497362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5367.046438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5367.052446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.140174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5368.151221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5368.198253] Lustre: lustre-MDT0000: new disk, initializing [ 5368.220872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5368.227459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5369.528670] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5373.638106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5373.688450] Lustre: 317325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5373.712688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5373.714526] Lustre: Skipped 1 previous similar message [ 5373.752437] Lustre: lustre-MDT0001: new disk, initializing [ 5373.787599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5373.801587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5373.804734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5375.216036] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5379.198227] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5381.179341] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5383.394652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5383.528502] Lustre: lustre-OST0000: new disk, initializing [ 5383.531719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5383.536361] Lustre: 319291:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5383.578607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.527280] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5389.260781] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5392.450789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5392.455807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5392.496204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5398.005081] Lustre: Mounted lustre-client [ 5398.363314] Lustre: Unmounted lustre-client [ 5402.593149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5402.596312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5402.600456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5403.107490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5403.107547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5403.112764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5407.577841] Lustre: server umount lustre-OST0000 complete [ 5409.760933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5409.763566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5409.770107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5409.772248] Lustre: Skipped 1 previous similar message [ 5413.346317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5413.346819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5415.071582] Lustre: server umount lustre-MDT0000 complete [ 5416.332503] LustreError: 320159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784127341 with bad export cookie 334074141796247413 [ 5416.337428] LustreError: 320159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5416.338202] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5416.470601] Lustre: server umount lustre-MDT0001 complete [ 5420.342770] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5421.409466] Key type lgssc unregistered [ 5446.112326] LNet: 321452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5446.114499] LNetError: 321452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5518.822416] LNet: Removed LNI 192.168.201.131@tcp [ 5519.150557] Key type .llcrypt unregistered [ 5519.151701] Key type ._llcrypt unregistered [ 5525.070951] Key type ._llcrypt registered [ 5525.072038] Key type .llcrypt registered [ 5525.108501] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 5531.472116] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5531.791253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5531.843591] alg: No test for adler32 (adler32-zlib) [ 5532.689142] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 5532.778156] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 5534.352100] Key type lgssc registered [ 5534.695957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5544.254699] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 10:57:48 (1784127468) [ 5546.382504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5546.387200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5547.475439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5547.485802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5547.531477] Lustre: lustre-MDT0000: new disk, initializing [ 5547.553544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5547.559862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5548.899247] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5553.014536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.054991] Lustre: 324647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5553.078730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5553.082202] Lustre: Skipped 1 previous similar message [ 5553.121441] Lustre: lustre-MDT0001: new disk, initializing [ 5553.153419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5553.165372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5553.168405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5554.491856] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5558.189102] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5560.058067] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5562.378326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5562.499358] Lustre: lustre-OST0000: new disk, initializing [ 5562.501082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5562.503582] Lustre: 326615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5562.534242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5564.089372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5564.092472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5564.129828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5564.456654] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5568.197093] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5574.627391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5574.628035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5574.628543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5574.629823] LustreError: Skipped 1 previous similar message [ 5574.634440] Lustre: Skipped 1 previous similar message [ 5575.630396] Lustre: server umount lustre-OST0000 complete [ 5578.721075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5578.725164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.729863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5578.731880] Lustre: Skipped 1 previous similar message [ 5578.978161] Lustre: server umount lustre-MDT0000 complete [ 5579.745939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.199278] LustreError: 324639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784127505 with bad export cookie 6161372266853365856 [ 5580.201942] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5580.202819] LustreError: 324639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5580.317176] Lustre: server umount lustre-MDT0001 complete [ 5583.869707] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5584.863288] Key type lgssc unregistered [ 5584.996465] LNet: 328521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5584.998763] LNetError: 328521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5585.007306] LNet: Removed LNI 192.168.201.131@tcp [ 5585.319541] Key type .llcrypt unregistered [ 5585.320692] Key type ._llcrypt unregistered [ 5593.051471] Key type ._llcrypt registered [ 5593.053036] Key type .llcrypt registered [ 5593.089341] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5593.461212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5593.504217] alg: No test for adler32 (adler32-zlib) [ 5594.365636] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 5594.457793] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 5596.032101] Key type lgssc registered [ 5596.394800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5598.937651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5598.943784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5600.084095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5601.394186] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5604.186585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5604.307492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5605.680847] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5608.031860] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5609.937525] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5612.189905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5612.304523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.225685] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5616.642611] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5630.669488] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5631.160447] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5632.550438] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.31@tcp (stopping) [ 5634.586481] Lustre: server umount lustre-MDT0000 complete [ 5635.042771] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.044762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5635.045590] LustreError: 332009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5635.045596] LustreError: 332009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5635.046365] Lustre: Skipped 2 previous similar messages [ 5635.788303] LustreError: 329797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784127560 with bad export cookie 1676026122980968159 [ 5635.792333] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5635.793359] LustreError: 329797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5637.663272] LustreError: 329796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5637.669766] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.31@tcp (stopping) [ 5637.670688] LustreError: 329796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5640.160874] LustreError: 329766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.161181] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5640.170222] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5641.964238] Lustre: server umount lustre-MDT0001 complete [ 5647.910299] Lustre: lustre-OST0000: Not available for connect from 192.168.201.31@tcp (stopping) [ 5649.372714] Lustre: server umount lustre-OST0000 complete [ 5652.245514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.627682] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5654.572881] Lustre: 333968:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.131@tcp [ 5655.269129] Lustre: server umount MGS complete [ 5658.153778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5658.306506] LustreError: 334539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.331590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5659.745936] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5662.625837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5662.756276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5664.094754] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5666.928689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5668.980918] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5671.508752] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5672.035208] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5673.737658] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5674.247342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5678.562377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.562602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5678.563199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5678.566753] Lustre: Skipped 1 previous similar message [ 5684.619695] Lustre: server umount lustre-OST0000 complete [ 5688.288732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5688.292129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5688.297917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5688.300524] Lustre: Skipped 3 previous similar messages [ 5691.993140] Lustre: server umount lustre-MDT0000 complete [ 5693.193327] LustreError: 334521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784127618 with bad export cookie 1676026122980968985 [ 5693.198180] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5693.199095] LustreError: 334521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5693.310919] Lustre: server umount lustre-MDT0001 complete [ 5697.195204] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5698.208916] Key type lgssc unregistered [ 5698.332381] LNet: 337969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5698.336304] LNetError: 337969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5698.347314] LNet: Removed LNI 192.168.201.131@tcp [ 5698.633166] Key type .llcrypt unregistered [ 5698.634423] Key type ._llcrypt unregistered [ 5704.660806] Key type ._llcrypt registered [ 5704.661992] Key type .llcrypt registered [ 5704.704155] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 5710.580926] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5710.980048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5710.992144] alg: No test for adler32 (adler32-zlib) [ 5711.841895] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 5711.917790] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 5713.496116] Key type lgssc registered [ 5713.801835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5724.445965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5724.452610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5725.559593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5725.569853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5725.606837] Lustre: lustre-MDT0000: new disk, initializing [ 5725.631257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5725.638582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5727.018821] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5731.268040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5731.314834] Lustre: 341101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5731.341860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5731.345046] Lustre: Skipped 1 previous similar message [ 5731.380811] Lustre: lustre-MDT0001: new disk, initializing [ 5731.412089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5731.425102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5731.428220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5732.740236] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5736.451927] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5738.337392] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5740.635482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5740.742402] Lustre: lustre-OST0000: new disk, initializing [ 5740.744093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5740.746915] Lustre: 343080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5740.776950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5742.791826] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5746.623659] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5748.029694] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5749.236817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5749.239666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5749.264471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5750.130382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5751.338536] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5751.407963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5754.338673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5754.340975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5754.341489] LustreError: Skipped 1 previous similar message [ 5754.342745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5754.348063] Lustre: Skipped 1 previous similar message [ 5758.099769] Lustre: server umount lustre-OST0000 complete [ 5759.457335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.457950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5759.462844] Lustre: Skipped 1 previous similar message [ 5759.464651] Lustre: Skipped 1 previous similar message [ 5764.577067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5764.579116] Lustre: Skipped 2 previous similar messages [ 5765.468531] Lustre: server umount lustre-MDT0000 complete [ 5766.715210] LustreError: 341093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784127691 with bad export cookie 6988102017622949011 [ 5766.719890] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5766.720056] LustreError: 341093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5766.831687] Lustre: server umount lustre-MDT0001 complete [ 5769.715741] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:01:34 (1784127694) [ 5775.091666] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 5778.412518] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5790.445832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.529630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5790.539328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5790.583715] Lustre: lustre-MDT0000: new disk, initializing [ 5790.605877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5790.612479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5792.019619] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5796.194367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5796.238893] Lustre: 347844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5796.262094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5796.264123] Lustre: Skipped 1 previous similar message [ 5796.301132] Lustre: lustre-MDT0001: new disk, initializing [ 5796.330465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5796.341523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5796.344603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5797.767581] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5801.390607] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5803.287544] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5805.565772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5805.663792] Lustre: lustre-OST0000: new disk, initializing [ 5805.665331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5805.667831] Lustre: 349814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5805.695778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5806.223159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5806.226467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5806.257913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5807.683325] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 5811.477480] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5821.924672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5821.925066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5821.926912] LustreError: Skipped 1 previous similar message [ 5821.931057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5826.832496] Lustre: server umount lustre-OST0000 complete [ 5832.161722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5832.161807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5832.162581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5832.162586] Lustre: Skipped 1 previous similar message [ 5832.164866] Lustre: Skipped 2 previous similar messages [ 5841.377592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5841.379715] Lustre: Skipped 4 previous similar messages [ 5842.400113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5842.472465] Lustre: server umount lustre-MDT0000 complete [ 5843.801932] LustreError: 348638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784127768 with bad export cookie 6988102017622949893 [ 5843.804909] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5843.806684] LustreError: 348638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5843.926776] Lustre: server umount lustre-MDT0001 complete [ 5847.228478] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5848.313622] Key type lgssc unregistered [ 5848.460383] LNet: 351962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5848.462677] LNetError: 351962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5848.471393] LNet: Removed LNI 192.168.201.131@tcp [ 5848.776970] Key type .llcrypt unregistered [ 5848.778033] Key type ._llcrypt unregistered [ 5879.728979] Key type ._llcrypt registered [ 5879.730364] Key type .llcrypt registered [ 5879.765040] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5879.863213] Key type .llcrypt unregistered [ 5879.864817] Key type ._llcrypt unregistered [ 5906.825503] Key type ._llcrypt registered [ 5906.826646] Key type .llcrypt registered [ 5906.864318] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5906.959240] Key type .llcrypt unregistered [ 5906.961212] Key type ._llcrypt unregistered [ 5918.155712] Key type ._llcrypt registered [ 5918.156741] Key type .llcrypt registered [ 5918.193809] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5918.290267] Key type .llcrypt unregistered [ 5918.291398] Key type ._llcrypt unregistered [ 5929.564163] Key type ._llcrypt registered [ 5929.565958] Key type .llcrypt registered [ 5929.609380] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5929.722188] Key type .llcrypt unregistered [ 5929.723926] Key type ._llcrypt unregistered [ 5957.271539] Key type ._llcrypt registered [ 5957.272567] Key type .llcrypt registered [ 5957.307355] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5957.417249] Key type .llcrypt unregistered [ 5957.419197] Key type ._llcrypt unregistered [ 5968.504708] Key type ._llcrypt registered [ 5968.506588] Key type .llcrypt registered [ 5968.546855] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5968.646314] Key type .llcrypt unregistered [ 5968.647374] Key type ._llcrypt unregistered [ 5979.673657] Key type ._llcrypt registered [ 5979.674666] Key type .llcrypt registered [ 5979.714930] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5979.809323] Key type .llcrypt unregistered [ 5979.810623] Key type ._llcrypt unregistered [ 5991.293913] Key type ._llcrypt registered [ 5991.294921] Key type .llcrypt registered [ 5991.342621] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5991.456296] Key type .llcrypt unregistered [ 5991.457443] Key type ._llcrypt unregistered [ 6003.689912] Key type ._llcrypt registered [ 6003.691185] Key type .llcrypt registered [ 6003.730388] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6003.838364] Key type .llcrypt unregistered [ 6003.840047] Key type ._llcrypt unregistered [ 6015.746136] Key type ._llcrypt registered [ 6015.747146] Key type .llcrypt registered [ 6015.791363] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6015.894322] Key type .llcrypt unregistered [ 6015.895387] Key type ._llcrypt unregistered [ 6037.222309] Key type ._llcrypt registered [ 6037.223402] Key type .llcrypt registered [ 6037.262683] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6037.367259] Key type .llcrypt unregistered [ 6037.368502] Key type ._llcrypt unregistered [ 6048.338969] Key type ._llcrypt registered [ 6048.340164] Key type .llcrypt registered [ 6048.378509] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6048.484925] Key type .llcrypt unregistered [ 6048.486167] Key type ._llcrypt unregistered [ 6060.024745] Key type ._llcrypt registered [ 6060.025698] Key type .llcrypt registered [ 6060.068197] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6060.181296] Key type .llcrypt unregistered [ 6060.182396] Key type ._llcrypt unregistered [ 6078.557327] Key type ._llcrypt registered [ 6078.558307] Key type .llcrypt registered [ 6078.597035] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 6084.992367] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6085.319733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6085.347035] alg: No test for adler32 (adler32-zlib) [ 6086.202562] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 6086.288777] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 6087.864157] Key type lgssc registered [ 6088.216578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6099.630164] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6099.636604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.721348] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 6100.731313] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 6100.768086] Lustre: tst250fs-MDT0000: new disk, initializing [ 6100.788704] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.794690] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6102.115842] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 6106.251598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6106.301510] Lustre: 371865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6106.329780] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 6106.331478] Lustre: Skipped 1 previous similar message [ 6106.369497] Lustre: tst250fs-MDT0001: new disk, initializing [ 6106.404395] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6106.417665] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6106.422415] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6107.791219] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 6110.091859] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6112.275526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6112.378105] Lustre: tst250fs-OST0000: new disk, initializing [ 6112.379738] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 6112.382065] Lustre: 373496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6112.414244] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.367607] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 6117.883349] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6117.886480] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6117.934323] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6118.481783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.549168] Lustre: tst250fs-OST0001: new disk, initializing [ 6118.550928] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 6118.553494] Lustre: 374369:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6118.587525] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6120.562809] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all [ 6123.087335] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 6124.855422] Lustre: 371870:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.131@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [ 6134.242666] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6134.245400] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6134.246356] Lustre: Skipped 3 previous similar messages [ 6134.248155] Lustre: Skipped 2 previous similar messages [ 6139.168249] Lustre: server umount tst250fs-MDT0000 complete [ 6140.332138] LustreError: 372662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784128065 with bad export cookie 15693693142671095151 [ 6140.335146] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6140.336294] LustreError: 372662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6140.444543] Lustre: server umount tst250fs-MDT0001 complete [ 6151.683697] Lustre: server umount tst250fs-OST0000 complete [ 6163.165448] Lustre: server umount tst250fs-OST0001 complete [ 6169.005745] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 6171.626403] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6181.678962] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:08:26 (1784128106) [ 6182.183270] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 6187.355703] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 6190.382334] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6200.492852] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6037 sec ========== 11:08:44 (1784128124) [ 6201.042228] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:08:45 (1784128125) === [ 6202.039427] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:08:46 (1784128126) === [ 6214.857488] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6215.786057] Key type lgssc unregistered [ 6215.915337] LNet: 383469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6215.917501] LNetError: 383469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6215.924257] LNet: Removed LNI 192.168.201.131@tcp [ 6216.239115] Key type .llcrypt unregistered [ 6216.240063] Key type ._llcrypt unregistered