[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 663944641 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002404] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009147] LSM: Security Framework initializing [ 0.010058] Yama: becoming mindful. [ 0.011033] SELinux: Initializing. [ 0.012057] *** VALIDATE selinux *** [ 0.021291] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025699] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] *** VALIDATE tmpfs *** [ 0.030446] *** VALIDATE proc *** [ 0.032017] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034287] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041703] debug: unmapping init [mem 0xffffffffa4859000-0xffffffffa4860fff] [ 0.043253] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044783] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045027] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048016] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050020] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.053266] rcu: Hierarchical SRCU implementation. [ 0.055626] smp: Bringing up secondary CPUs ... [ 0.056622] x86: Booting SMP configuration: [ 0.057029] .... node #0, CPUs: #1 #2 #3 [ 0.068155] smp: Brought up 1 node, 4 CPUs [ 0.070016] smpboot: Max logical packages: 1 [ 0.071022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.100770] node 0 deferred pages initialised in 27ms [ 0.104297] devtmpfs: initialized [ 0.105276] x86/mm: Memory block size: 128MB [ 0.108519] gcov: version magic: 0x41383552 [ 0.110223] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.111086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.112277] pinctrl core: initialized pinctrl subsystem [ 0.113216] [ 0.113941] ************************************************************* [ 0.114019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.115017] ** ** [ 0.116015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.117019] ** ** [ 0.118016] ** This means that this kernel is built to expose internal ** [ 0.119020] ** IOMMU data structures, which may compromise security on ** [ 0.120014] ** your system. ** [ 0.121014] ** ** [ 0.122016] ** If you see this message and you are not debugging the ** [ 0.123016] ** kernel, report this immediately to your vendor! ** [ 0.124014] ** ** [ 0.125018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.126015] ************************************************************* [ 0.128000] NET: Registered protocol family 16 [ 0.129782] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.130115] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.131071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.132904] cpuidle: using governor menu [ 0.135029] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.137789] PCI: Using configuration type 1 for base access [ 0.140153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.149338] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.150018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.152061] cryptd: max_cpu_qlen set to 1000 [ 0.154194] ACPI: Added _OSI(Module Device) [ 0.155011] ACPI: Added _OSI(Processor Device) [ 0.157011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.158012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.165426] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.172631] ACPI: Interpreter enabled [ 0.174073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.176011] ACPI: Using IOAPIC for interrupt routing [ 0.178187] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.182594] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.196667] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.199046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.201072] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.205096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.210711] acpiphp: Slot [2] registered [ 0.212173] acpiphp: Slot [5] registered [ 0.214201] acpiphp: Slot [6] registered [ 0.215113] acpiphp: Slot [7] registered [ 0.217121] acpiphp: Slot [8] registered [ 0.218101] acpiphp: Slot [9] registered [ 0.220156] acpiphp: Slot [10] registered [ 0.221213] acpiphp: Slot [3] registered [ 0.223112] acpiphp: Slot [4] registered [ 0.224188] acpiphp: Slot [11] registered [ 0.225081] acpiphp: Slot [12] registered [ 0.226000] acpiphp: Slot [13] registered [ 0.226000] acpiphp: Slot [14] registered [ 0.228097] acpiphp: Slot [15] registered [ 0.229140] acpiphp: Slot [16] registered [ 0.231159] acpiphp: Slot [17] registered [ 0.233100] acpiphp: Slot [18] registered [ 0.234096] acpiphp: Slot [19] registered [ 0.236228] acpiphp: Slot [20] registered [ 0.238244] acpiphp: Slot [21] registered [ 0.239094] acpiphp: Slot [22] registered [ 0.241134] acpiphp: Slot [23] registered [ 0.242147] acpiphp: Slot [24] registered [ 0.244112] acpiphp: Slot [25] registered [ 0.246147] acpiphp: Slot [26] registered [ 0.247194] acpiphp: Slot [27] registered [ 0.249113] acpiphp: Slot [28] registered [ 0.251118] acpiphp: Slot [29] registered [ 0.252121] acpiphp: Slot [30] registered [ 0.254113] acpiphp: Slot [31] registered [ 0.256085] PCI host bridge to bus 0000:00 [ 0.257017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.260024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.262020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.265027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.267024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.270030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.272273] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.276131] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.279534] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.289847] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.296119] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.299018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.301015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.303025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.307678] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.310889] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.314048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.318021] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.324017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.340018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.346016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.354181] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.418000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.424022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.439816] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.450021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.461020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.509022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.520000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.520000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.520000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.529021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.538782] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.546018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.556018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.576022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.590013] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.598019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.606020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.625027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.635115] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.641017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.646021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.663017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.673537] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.677529] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.679385] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.681499] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.682213] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.683000] iommu: Default domain type: Passthrough [ 0.683000] SCSI subsystem initialized [ 0.683000] ACPI: bus type USB registered [ 0.683000] usbcore: registered new interface driver usbfs [ 0.683000] usbcore: registered new interface driver hub [ 0.683000] usbcore: registered new device driver usb [ 0.683000] pps_core: LinuxPPS API ver. 1 registered [ 0.683000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.683000] PTP clock support registered [ 0.683193] EDAC MC: Ver: 3.0.0 [ 0.684000] PCI: Using ACPI for IRQ routing [ 0.684000] NetLabel: Initializing [ 0.684000] NetLabel: domain hash size = 128 [ 0.684000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.684000] NetLabel: unlabeled traffic allowed by default [ 0.684043] vgaarb: loaded [ 0.685000] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.685000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.693835] clocksource: Switched to clocksource kvm-clock [ 0.830223] VFS: Disk quotas dquot_6.6.0 [ 0.832210] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.835262] *** VALIDATE ramfs *** [ 0.836748] *** VALIDATE hugetlbfs *** [ 0.838606] pnp: PnP ACPI init [ 0.841627] pnp: PnP ACPI: found 6 devices [ 0.860331] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.864401] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.867178] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.869596] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.872185] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.874964] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.877919] NET: Registered protocol family 2 [ 0.880836] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.887090] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.891442] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.897129] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.900410] TCP: Hash tables configured (established 65536 bind 65536) [ 0.903544] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.906446] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.909189] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.912191] NET: Registered protocol family 1 [ 0.914701] RPC: Registered named UNIX socket transport module. [ 0.917381] RPC: Registered udp transport module. [ 0.919608] RPC: Registered tcp transport module. [ 0.921924] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.925124] NET: Registered protocol family 44 [ 0.927259] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.929882] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.932490] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.935532] PCI: CLS 0 bytes, default 64 [ 0.937644] Unpacking initramfs... [ 3.739362] debug: unmapping init [mem 0xffff8942bcc54000-0xffff8942bffbffff] [ 3.746591] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.749508] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.757236] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.788710] Initialise system trusted keyrings [ 4.790803] Key type blacklist registered [ 4.792925] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.802829] zbud: loaded [ 4.806739] *** VALIDATE nfs *** [ 4.808053] *** VALIDATE nfs4 *** [ 4.809968] pstore: using deflate compression [ 4.814507] Platform Keyring initialized [ 4.982462] NET: Registered protocol family 38 [ 4.984273] Key type asymmetric registered [ 4.986341] Asymmetric key parser 'x509' registered [ 4.988332] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.991267] io scheduler mq-deadline registered [ 4.993537] io scheduler kyber registered [ 4.995717] io scheduler bfq registered [ 4.997910] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.001793] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.005395] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.008587] ACPI: Power Button [PWRF] [ 5.017507] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.034221] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.056548] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.066191] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.087807] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.118733] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.151790] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.159485] Non-volatile memory driver v1.3 [ 5.161910] Linux agpgart interface v0.103 [ 5.200258] virtio_blk virtio1: [vda] 134848 512-byte logical blocks (69.0 MB/65.8 MiB) [ 5.203268] vda: detected capacity change from 0 to 69042176 [ 5.218868] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.222567] vdb: detected capacity change from 0 to 1073741824 [ 5.247303] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.251898] vdc: detected capacity change from 0 to 2621440000 [ 5.275555] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.280679] vdd: detected capacity change from 0 to 2621440000 [ 5.309382] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.313506] vde: detected capacity change from 0 to 4294967296 [ 5.334217] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.338287] vdf: detected capacity change from 0 to 4294967296 [ 5.347364] libphy: Fixed MDIO Bus: probed [ 5.370985] usbcore: registered new interface driver usbserial_generic [ 5.374167] usbserial: USB Serial support registered for generic [ 5.377225] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.382905] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.385413] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.388406] mousedev: PS/2 mouse device common for all mice [ 5.393768] rtc_cmos 00:05: RTC can wake from S4 [ 5.397675] rtc_cmos 00:05: registered as rtc0 [ 5.400226] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.404718] intel_pstate: CPU model not supported [ 5.407381] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.412588] hid: raw HID events driver (C) Jiri Kosina [ 5.418988] usbcore: registered new interface driver usbhid [ 5.423733] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.424873] usbhid: USB HID core driver [ 5.425836] drop_monitor: Initializing network drop monitor service [ 5.434154] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.438150] Initializing XFRM netlink socket [ 5.442337] NET: Registered protocol family 10 [ 5.449279] Segment Routing with IPv6 [ 5.451176] NET: Registered protocol family 17 [ 5.453845] mpls_gso: MPLS GSO support [ 5.460471] RAS: Correctable Errors collector initialized. [ 5.462939] AVX version of gcm_enc/dec engaged. [ 5.464686] AES CTR mode by8 optimization enabled [ 5.588178] sched_clock: Marking stable (5588153011, 0)->(6984363556, -1396210545) [ 5.642144] registered taskstats version 1 [ 5.648276] Loading compiled-in X.509 certificates [ 5.650078] zswap: loaded using pool lzo/zbud [ 5.728676] Key type big_key registered [ 5.747882] Key type encrypted registered [ 5.750129] ima: No TPM chip found, activating TPM-bypass! [ 5.752967] ima: Allocated hash algorithm: sha1 [ 5.754671] ima: No architecture policies found [ 5.756905] evm: Initialising EVM extended attributes: [ 5.759907] evm: security.selinux [ 5.761685] evm: security.ima [ 5.763173] evm: security.capability [ 5.764783] evm: HMAC attrs: 0x1 [ 5.768159] rtc_cmos 00:05: setting system clock to 2026-06-01 14:23:53 UTC (1780323833) [ 5.776515] debug: unmapping init [mem 0xffffffffa5803000-0xffffffffa59fffff] [ 5.781789] debug: unmapping init [mem 0xffffffffa4582000-0xffffffffa4858fff] [ 5.792222] Write protecting the kernel read-only data: 28672k [ 5.799060] debug: unmapping init [mem 0xffffffffa2c03000-0xffffffffa2dfffff] [ 5.802844] debug: unmapping init [mem 0xffffffffa3514000-0xffffffffa35fffff] [ 5.880785] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.897642] systemd[1]: Detected virtualization kvm. [ 5.905065] systemd[1]: Detected architecture x86-64. [ 5.907652] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.954829] systemd[1]: No hostname configured. [ 5.961095] systemd[1]: Set hostname to . [ 5.969575] random: systemd: uninitialized urandom read (16 bytes read) [ 5.979154] systemd[1]: Initializing machine ID from random generator. [ 6.119595] random: ln: uninitialized urandom read (6 bytes read) [ 6.347707] random: systemd: uninitialized urandom read (16 bytes read) [ 6.356847] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.375568] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.388277] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.866355] device-mapper: uevent: version 1.0.3 [ 7.868668] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.178952] virtio_net virtio0 ens2: renamed from eth0 [ 9.219826] random: fast init done [ 9.404242] scsi host0: ata_piix [ 9.424151] scsi host1: ata_piix [ 9.496461] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.502571] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.277922] random: crng init done [ 14.280180] random: 7 urandom warning(s) missed due to ratelimiting [ 16.194923] dracut-initqueue[599]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.651077] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.670722] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.740331] SELinux: Disabled at runtime. [ 21.853247] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.863315] systemd[1]: Detected virtualization kvm. [ 21.865441] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.555438] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.559872] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.584517] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.595958] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.604905] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.621434] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.648045] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 23.823949] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.725886] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.624154] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.662722] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.386841] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.472661] EDAC sbridge: Ver: 1.1.2 [ 29.628881] Key type dns_resolver registered [ 30.305689] NFS: Registering the id_resolver key type [ 30.310195] Key type id_resolver registered [ 30.312961] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg215-server login: [ 74.774889] libcfs: loading out-of-tree module taints kernel. [ 74.804276] Key type ._llcrypt registered [ 74.806123] Key type .llcrypt registered [ 74.868217] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 87.390547] hrtimer: interrupt took 2572166 ns [ 87.551814] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 88.460347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.470137] alg: No test for adler32 (adler32-zlib) [ 89.638243] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 90.164575] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 91.911187] Key type lgssc registered [ 93.036435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 103.409119] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 130.198726] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 138.250297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 138.261233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 139.395878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 139.423176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 139.487273] Lustre: lustre-MDT0000: new disk, initializing [ 139.533092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.543419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 141.999133] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.326065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.431436] Lustre: 6508:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 151.448267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 151.450318] Lustre: Skipped 1 previous similar message [ 151.513370] Lustre: lustre-MDT0001: new disk, initializing [ 151.553355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 151.570340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 151.576200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 154.223558] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.783730] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 164.672513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 164.832739] Lustre: lustre-OST0000: new disk, initializing [ 164.835418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 164.840445] Lustre: 8413:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 164.885633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.778960] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.511552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 170.520466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 170.552758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 179.520989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 179.607366] Lustre: lustre-OST0001: new disk, initializing [ 179.609727] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 179.613419] Lustre: 9467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 179.676447] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.374192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 180.381656] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 180.436187] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 184.456143] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.891683] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 199.557550] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 204.483382] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing check_logdir /tmp/testlogs/ [ 207.856584] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing yml_node [ 211.332249] Lustre: DEBUG MARKER: Client: 2.17.53.29 [ 213.105197] Lustre: DEBUG MARKER: MDS: 2.17.53.29 [ 214.686974] Lustre: DEBUG MARKER: OSS: 2.17.53.29 [ 215.659426] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jun 1 10:27:21 EDT 2026 [ 229.372401] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 230.668776] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 247.266621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 247.272672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 247.281202] Lustre: Skipped 2 previous similar messages [ 247.299083] Lustre: Skipped 3 previous similar messages [ 250.729338] Lustre: server umount lustre-MDT0000 complete [ 253.624808] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324081 with bad export cookie 11649647528358384767 [ 253.628503] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 253.638788] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 257.503893] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.506049] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 257.535592] Lustre: Skipped 1 previous similar message [ 257.546918] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 257.561207] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 258.527986] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.539099] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 259.879391] Lustre: server umount lustre-MDT0001 complete [ 262.971435] Lustre: server umount lustre-OST0000 complete [ 266.209611] Lustre: server umount lustre-OST0001 complete [ 271.048831] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 278.054856] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 310.693365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 310.948057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 310.990516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 311.104757] Lustre: lustre-MDT0000: new disk, initializing [ 311.171255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 311.200361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 314.815684] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 322.457265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 322.596274] Lustre: 14971:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 322.639853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 322.652361] Lustre: Skipped 1 previous similar message [ 322.715661] Lustre: lustre-MDT0001: new disk, initializing [ 322.803897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 322.849906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 322.872937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 326.238992] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 333.246488] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 336.099188] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 341.348639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 341.625387] Lustre: lustre-OST0000: new disk, initializing [ 341.628873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 341.634811] Lustre: 16716:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 341.717291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 342.975460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 342.992138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 343.056850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 345.934713] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 351.604435] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 355.697735] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 355.907824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 359.567918] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 359.723150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 363.496123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.506932] Lustre: Skipped 1 previous similar message [ 363.509206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 363.510487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 363.516683] LustreError: Skipped 1 previous similar message [ 363.520270] Lustre: Skipped 3 previous similar messages [ 367.605029] Lustre: server umount lustre-OST0000 complete [ 373.734071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.748923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 373.755972] Lustre: Skipped 2 previous similar messages [ 373.770095] Lustre: Skipped 1 previous similar message [ 376.784162] Lustre: server umount lustre-MDT0000 complete [ 378.851667] LustreError: 14985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.872774] LustreError: 14985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 380.122513] LustreError: 15751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324207 with bad export cookie 11649647528358386300 [ 380.124548] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.127679] LustreError: 15751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 380.647158] Lustre: server umount lustre-MDT0001 complete [ 385.831633] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:30:11 (1780324211) [ 400.939564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.263771] LustreError: 19443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.326234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.082100] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 406.503849] LustreError: 19444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.704428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.809322] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 421.208892] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 424.590415] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 426.260571] Lustre: Failing over lustre-MDT0000 [ 426.494959] Lustre: server umount lustre-MDT0000 complete [ 427.490401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 427.493385] LustreError: 19444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 427.498679] Lustre: Skipped 1 previous similar message [ 427.519236] LustreError: 19444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 443.873979] Lustre: 3663:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780324255/real 1780324255] req@ffff894336450380 x1866804931732736/t0(0) o400->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780324271 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 443.905446] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.923628] LustreError: 20153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 443.937394] LustreError: 20153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 444.119075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.378082] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 457.381104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 457.405910] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 457.941548] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 463.956041] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 467.945897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 467.952631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.966693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 473.823886] Lustre: server umount lustre-MDT0000 complete [ 477.465206] LustreError: 19424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324305 with bad export cookie 11649647528358387714 [ 477.475063] LustreError: 19424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 477.477450] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.763203] Lustre: server umount lustre-MDT0001 complete [ 484.807691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.208256] LustreError: 22437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.221581] LustreError: 22437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 485.264297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.267384] Lustre: Skipped 2 previous similar messages [ 488.352572] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 495.228467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.793916] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 503.250497] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 506.583068] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 508.374536] Lustre: Failing over lustre-MDT0000 [ 508.385286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.386580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 508.393378] Lustre: Skipped 2 previous similar messages [ 508.413485] Lustre: Skipped 3 previous similar messages [ 508.703549] Lustre: server umount lustre-MDT0000 complete [ 526.597936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.753287] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 531.190545] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 532.452136] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 532.463947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 532.483370] Lustre: Skipped 1 previous similar message [ 532.508907] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 537.515864] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 542.691843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 542.700105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.713577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 546.980780] Lustre: server umount lustre-MDT0000 complete [ 549.753672] LustreError: 23781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324377 with bad export cookie 11649647528358388652 [ 549.758111] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.765444] LustreError: 23781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 550.010842] Lustre: server umount lustre-MDT0001 complete [ 556.904355] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:33:02 (1780324382) [ 571.070992] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 579.519600] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 610.995377] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 619.186979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.396828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 619.418723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 619.481470] Lustre: lustre-MDT0000: new disk, initializing [ 619.545626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.549562] Lustre: Skipped 2 previous similar messages [ 619.560117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 622.986668] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 630.912485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.995569] Lustre: 28415:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 631.195218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 634.251481] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 639.468798] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 641.718468] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 645.747808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 645.967484] Lustre: 30155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 647.434656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 647.460663] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 647.496824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 647.514574] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 650.253203] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 655.986551] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 661.408257] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 673.258940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.260587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 673.262630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 673.267080] Lustre: Skipped 2 previous similar messages [ 673.282084] Lustre: Skipped 3 previous similar messages [ 679.125533] Lustre: server umount lustre-OST0000 complete [ 682.468892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 682.476837] LustreError: Skipped 1 previous similar message [ 687.978832] Lustre: server umount lustre-MDT0000 complete [ 688.609583] LustreError: 28426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 688.617229] LustreError: 28426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 690.574069] LustreError: 28407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324518 with bad export cookie 11649647528358389058 [ 690.582417] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 690.587377] LustreError: 28407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 690.875509] Lustre: server umount lustre-MDT0001 complete [ 697.334847] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 699.306242] Key type lgssc unregistered [ 699.543729] LNet: 31984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.551417] LNetError: 31984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 699.574333] LNet: Removed LNI 192.168.202.115@tcp [ 700.127138] Key type .llcrypt unregistered [ 700.128808] Key type ._llcrypt unregistered [ 716.153971] Key type ._llcrypt registered [ 716.155881] Key type .llcrypt registered [ 716.252307] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 717.079379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 717.103801] alg: No test for adler32 (adler32-zlib) [ 718.041395] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 718.235221] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 719.919127] Key type lgssc registered [ 720.685652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 725.616267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 725.627934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.918844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.494311] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 734.621041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 734.844528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 737.289621] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 740.685305] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 743.062102] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 747.707935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 748.052440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.467459] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 754.593061] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 759.346956] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 767.700281] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:36:33 (1780324593) [ 768.919933] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 770.087373] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 774.433670] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:36:40 (1780324600) [ 779.492875] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:36:45 (1780324605) [ 780.254113] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 785.471513] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 785.474496] Lustre: Skipped 2 previous similar messages [ 786.607852] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:36:52 (1780324612) [ 791.750921] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 791.756099] Lustre: Skipped 4 previous similar messages [ 800.343972] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:37:06 (1780324626) [ 801.895377] Lustre: *** cfs_fail_loc=131b, val=2*** [ 804.032806] Lustre: *** cfs_fail_loc=131b, val=2*** [ 808.917191] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:37:15 (1780324635) [ 809.937750] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 809.941308] Lustre: Skipped 9 previous similar messages [ 816.387346] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:37:22 (1780324642) [ 823.776475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 823.781515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.793554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 825.824557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 825.830123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 825.832437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 825.843111] Lustre: Skipped 1 previous similar message [ 829.673703] Lustre: server umount lustre-OST0000 complete [ 831.968038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 831.971376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 831.976708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 836.065575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.066388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 836.078345] Lustre: Skipped 1 previous similar message [ 837.575083] Lustre: server umount lustre-MDT0000 complete [ 839.576335] LustreError: 35944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324667 with bad export cookie 13012665711812945509 [ 839.577405] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 839.583246] LustreError: 35944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 839.755341] Lustre: server umount lustre-MDT0001 complete [ 845.019110] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 846.599161] Key type lgssc unregistered [ 846.788946] LNet: 40839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.792848] LNetError: 40839:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.805422] LNet: Removed LNI 192.168.202.115@tcp [ 847.231125] Key type .llcrypt unregistered [ 847.233467] Key type ._llcrypt unregistered [ 856.390066] Key type ._llcrypt registered [ 856.392333] Key type .llcrypt registered [ 856.455835] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 865.316913] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 865.903831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 865.924939] alg: No test for adler32 (adler32-zlib) [ 866.852909] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 866.986544] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 868.639173] Key type lgssc registered [ 869.271378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 887.471914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 887.480983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.599923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 888.618178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 888.661154] Lustre: lustre-MDT0000: new disk, initializing [ 888.726184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 888.735866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 890.961346] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 896.758991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.820098] Lustre: 43933:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 896.863333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 896.866394] Lustre: Skipped 1 previous similar message [ 896.914297] Lustre: lustre-MDT0001: new disk, initializing [ 896.964040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 896.989610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 896.996082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 898.847358] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 903.273487] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 905.130602] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 908.969715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 909.095735] Lustre: lustre-OST0000: new disk, initializing [ 909.098724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 909.102434] Lustre: 45675:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 909.144721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 911.645099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 911.655963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 911.725484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 912.202872] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 917.113288] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 920.079689] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 920.242972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 922.634258] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 922.769437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 927.204237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 927.205452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.207570] LustreError: Skipped 1 previous similar message [ 927.211262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 927.221855] Lustre: Skipped 1 previous similar message [ 930.093807] Lustre: server umount lustre-OST0000 complete [ 932.322110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.324400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 932.329293] Lustre: Skipped 1 previous similar message [ 932.332683] Lustre: Skipped 1 previous similar message [ 937.442404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 937.453636] Lustre: Skipped 2 previous similar messages [ 938.286554] Lustre: server umount lustre-MDT0000 complete [ 940.611485] LustreError: 43927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324768 with bad export cookie 14319986029969677950 [ 940.613273] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 940.619917] LustreError: 43927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 940.806120] Lustre: server umount lustre-MDT0001 complete [ 945.658509] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:39:31 (1780324771) [ 949.315753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.544056] LustreError: 47984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.553796] LustreError: 47984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 949.582273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.611322] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 954.850333] LustreError: 47985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.401783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.617386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 958.591577] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 961.424287] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 963.335958] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 967.148089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 967.380434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 970.395137] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 972.855824] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 986.734648] LustreError: 50196:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 987.285313] Lustre: Modifying parameter general.timeout=1 in log params [ 988.674086] Lustre: Modifying parameter general.timeout=4 in log params [ 988.676589] Lustre: Skipped 2 previous similar messages [ 990.815299] Lustre: Modifying parameter general.timeout=9 in log params [ 990.818680] Lustre: Skipped 4 previous similar messages [ 994.991512] Lustre: Modifying parameter general.timeout=20 in log params [ 994.994404] Lustre: Skipped 10 previous similar messages [ 1003.361712] Lustre: Modifying parameter general.timeout=38 in log params [ 1003.364609] Lustre: Skipped 17 previous similar messages [ 1019.799633] Lustre: Modifying parameter general.timeout=73 in log params [ 1019.802407] Lustre: Skipped 34 previous similar messages [ 1051.070644] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:41:17 (1780324877) [ 1051.978220] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1051.982602] Lustre: Skipped 58 previous similar messages [ 1052.422607] Lustre: 57105:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1054.086056] Lustre: 57301:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1054.091529] Lustre: 57301:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1056.682583] Lustre: 57596:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1056.687111] Lustre: 57596:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1070.624379] Lustre: 58943:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1070.631200] LustreError: 58943:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1073.014980] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:41:39 (1780324899) [ 1079.264229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.270589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.277029] Lustre: Skipped 2 previous similar messages [ 1079.280051] Lustre: Skipped 1 previous similar message [ 1084.389412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.395759] Lustre: Skipped 1 previous similar message [ 1085.049547] Lustre: server umount lustre-MDT0000 complete [ 1087.160324] LustreError: 47964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324914 with bad export cookie 14319986029969678832 [ 1087.161597] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1087.166487] LustreError: 47964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1087.347711] Lustre: server umount lustre-MDT0001 complete [ 1099.654303] Lustre: server umount lustre-OST0000 complete [ 1113.888690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1114.063397] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1114.223324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.221420] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1121.264308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.330866] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1121.346543] Lustre: 61469:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1123.826603] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1126.748218] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1128.741865] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1132.217363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1132.318182] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1135.321690] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1138.645877] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1144.963206] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1144.967730] Lustre: Skipped 21 previous similar messages [ 1144.990339] Lustre: 63697:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1150.436286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1150.437037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.440579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1150.445128] Lustre: Skipped 1 previous similar message [ 1150.449657] Lustre: Skipped 1 previous similar message [ 1153.771928] Lustre: server umount lustre-OST0000 complete [ 1157.601070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1157.607392] LustreError: Skipped 1 previous similar message [ 1157.614530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.763923] Lustre: server umount lustre-MDT0000 complete [ 1163.663880] LustreError: 61460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780324991 with bad export cookie 14319986029969681310 [ 1163.667137] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1163.671768] LustreError: 61460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1163.801795] Lustre: server umount lustre-MDT0001 complete [ 1169.605156] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 1171.126918] Key type lgssc unregistered [ 1171.325474] LNet: 64890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.328774] LNetError: 64890:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.337520] LNet: Removed LNI 192.168.202.115@tcp [ 1171.882192] Key type .llcrypt unregistered [ 1171.885323] Key type ._llcrypt unregistered [ 1184.611091] Key type ._llcrypt registered [ 1184.615083] Key type .llcrypt registered [ 1184.672880] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1185.154215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1185.187107] alg: No test for adler32 (adler32-zlib) [ 1186.087941] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1186.219171] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1187.839183] Key type lgssc registered [ 1188.364658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1192.205881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1192.217351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.463424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.251136] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1199.756718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.037608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1202.104885] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1205.025510] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1206.640040] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1210.113530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1210.420353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1213.917812] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1217.091533] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1227.171886] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1230.817296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1230.824118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.836673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.889819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.895080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.903168] Lustre: Skipped 1 previous similar message [ 1236.959269] Lustre: server umount lustre-MDT0000 complete [ 1238.721886] LustreError: 66243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325066 with bad export cookie 4515732121814050958 [ 1238.725170] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1238.726447] LustreError: 66243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1238.864637] Lustre: server umount lustre-MDT0001 complete [ 1251.174668] Lustre: server umount lustre-OST0000 complete [ 1264.419597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.581927] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1264.600124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1264.789697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.725832] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1271.093227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.157706] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1271.171802] Lustre: 70789:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1271.403444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1273.483948] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1276.415690] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1278.236618] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1281.617124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1281.713109] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1281.911581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.920717] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1288.171591] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1289.008427] Lustre: 73024:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1290.170127] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1292.768792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1292.769748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.773734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1292.779401] Lustre: Skipped 1 previous similar message [ 1292.786493] Lustre: Skipped 2 previous similar messages [ 1294.467328] Lustre: server umount lustre-OST0000 complete [ 1296.866450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1296.873045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.889596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.894689] Lustre: Skipped 1 previous similar message [ 1303.008723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1303.014043] Lustre: Skipped 3 previous similar messages [ 1311.199145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1311.286187] Lustre: server umount lustre-MDT0000 complete [ 1313.248848] LustreError: 70802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.260056] LustreError: 70802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1313.374233] LustreError: 72429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325141 with bad export cookie 4515732121814051777 [ 1313.377376] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1313.379769] LustreError: 72429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1313.594754] Lustre: server umount lustre-MDT0001 complete [ 1319.209947] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 1320.688482] Key type lgssc unregistered [ 1320.847739] LNet: 74216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1320.851650] LNetError: 74216:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1320.867263] LNet: Removed LNI 192.168.202.115@tcp [ 1321.259314] Key type .llcrypt unregistered [ 1321.260837] Key type ._llcrypt unregistered [ 1333.513387] Key type ._llcrypt registered [ 1333.515212] Key type .llcrypt registered [ 1333.575477] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1334.137292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1334.148146] alg: No test for adler32 (adler32-zlib) [ 1335.059280] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1335.190555] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1336.799269] Key type lgssc registered [ 1337.453831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1341.205705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1341.217733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.472843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.510850] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1348.661576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.977832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1350.754311] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1353.335102] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1355.162126] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1358.553645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1358.819020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.567466] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1364.318396] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1371.986468] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1372.778442] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1372.781309] Lustre: Skipped 1 previous similar message [ 1373.992593] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1373.998387] Lustre: Skipped 2 previous similar messages [ 1376.072675] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1376.075684] Lustre: Skipped 4 previous similar messages [ 1380.210404] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1380.216382] Lustre: Skipped 9 previous similar messages [ 1388.242669] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1388.248422] Lustre: Skipped 18 previous similar messages [ 1404.601294] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1404.604939] Lustre: Skipped 36 previous similar messages [ 1418.397353] Lustre: 83326:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1419.609255] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:47:25 (1780325245) [ 1420.346421] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1421.275972] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:47:27 (1780325247) [ 1424.353926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.353942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1424.360958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1424.365760] Lustre: Skipped 1 previous similar message [ 1428.703755] Lustre: server umount lustre-OST0000 complete [ 1431.008284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1431.011783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.017351] Lustre: Skipped 1 previous similar message [ 1431.020122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.592922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.594110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.656359] Lustre: server umount lustre-MDT0000 complete [ 1438.314414] LustreError: 75571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325266 with bad export cookie 13518278808569429664 [ 1438.318233] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1438.322757] LustreError: 75571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1438.464681] Lustre: server umount lustre-MDT0001 complete [ 1443.072639] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 1444.323362] Key type lgssc unregistered [ 1444.468612] LNet: 84554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.473320] LNetError: 84554:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.483247] LNet: Removed LNI 192.168.202.115@tcp [ 1444.924886] Key type .llcrypt unregistered [ 1444.926636] Key type ._llcrypt unregistered [ 1446.675971] Key type ._llcrypt registered [ 1446.677418] Key type .llcrypt registered [ 1446.733779] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1448.424703] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules [ 1448.859816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1448.888306] alg: No test for adler32 (adler32-zlib) [ 1449.774082] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1449.872492] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1451.471190] Key type lgssc registered [ 1452.062332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1452.611094] LustreError: 85675:0:(lu_object.c:1479:lu_context_key_register()) cfs_race id 60d sleeping [ 1462.257661] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1466.644305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1473.741358] LustreError: 85675:0:(lu_object.c:1479:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1473.758059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.057108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.738597] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1480.688898] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:48:27 (1780325307) [ 1482.796442] LustreError: 86713:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1482.801939] LustreError: 86713:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1482.806709] LustreError: 86713:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1482.929406] Lustre: server umount lustre-MDT0000 complete [ 1489.511408] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 1490.971277] Key type lgssc unregistered [ 1491.151870] LNet: 88169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.155301] LNetError: 88169:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.170175] LNet: Removed LNI 192.168.202.115@tcp [ 1491.577190] Key type .llcrypt unregistered [ 1491.579735] Key type ._llcrypt unregistered [ 1500.136789] Key type ._llcrypt registered [ 1500.138595] Key type .llcrypt registered [ 1500.197968] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1500.612920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1500.659231] alg: No test for adler32 (adler32-zlib) [ 1501.528345] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1501.625284] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1503.215172] Key type lgssc registered [ 1503.736605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1507.756429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1507.778606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.080839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1510.837705] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1515.343545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.703060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1518.007855] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1521.001566] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1522.775636] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1525.903961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1526.210400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1529.224667] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1531.699458] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1568.659974] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:49:55 (1780325395) [ 1569.480862] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1570.398642] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:49:56 (1780325396) [ 1571.808092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1571.810864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.816313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.418060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.418271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.436222] Lustre: Skipped 1 previous similar message [ 1577.977741] Lustre: server umount lustre-MDT0000 complete [ 1579.729691] LustreError: 89327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325407 with bad export cookie 7436940097374995662 [ 1579.732919] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1579.735213] LustreError: 89327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1579.885895] Lustre: server umount lustre-MDT0001 complete [ 1592.091924] Lustre: server umount lustre-OST0000 complete [ 1598.193893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.446880] LustreError: 93295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1598.461314] LustreError: 93295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1598.548711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.446894] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1603.555871] LustreError: 93296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1604.775226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.111676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1607.091891] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1609.922828] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1611.708350] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1618.572194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1618.675602] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1618.679179] LustreError: 93284:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1618.694811] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1618.699374] LustreError: 95150:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1618.702432] LustreError: 95150:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1618.705555] LustreError: 95150:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1618.740125] Lustre: server umount lustre-OST0000 complete [ 1618.742548] LustreError: 95150:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1623.277228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1623.388745] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1623.392930] LustreError: 93284:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1623.397058] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1623.401152] LustreError: 95611:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1623.404281] LustreError: 95611:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1623.407073] LustreError: 95611:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1623.439041] Lustre: server umount lustre-OST0000 complete [ 1623.441760] LustreError: 95611:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1628.488067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1628.595986] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1628.601949] Lustre: Found index 0 for lustre-OST0000, updating log [ 1628.606527] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1628.642841] Lustre: lustre-OST0000: new disk, initializing [ 1628.645280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1628.649048] Lustre: 96146:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1628.773780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.233451] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1633.029025] Lustre: Failing over lustre-OST0000 [ 1633.113770] Lustre: server umount lustre-OST0000 complete [ 1635.297144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.301295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1635.305966] Lustre: Skipped 1 previous similar message [ 1635.308719] Lustre: Skipped 2 previous similar messages [ 1640.416852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1640.420732] Lustre: Skipped 2 previous similar messages [ 1641.273469] Lustre: server umount lustre-MDT0000 complete [ 1643.417086] LustreError: 93275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325471 with bad export cookie 7436940097374998392 [ 1643.419069] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1643.421609] LustreError: 93275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1653.808468] Lustre: server umount lustre-MDT0001 complete [ 1658.085243] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:51:24 (1780325484) [ 1664.976304] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1669.552992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.753677] LustreError: 98456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.804896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1671.711314] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1675.232145] LustreError: 98457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1675.368986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.492695] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1678.868915] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1682.080221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1682.416706] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1685.215922] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1685.482715] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1685.495363] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1685.495538] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1689.391439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1689.442481] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1689.483448] Lustre: lustre-OST0001: new disk, initializing [ 1689.486162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1689.490186] Lustre: 100722:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1692.210161] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1692.372665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1692.377373] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1692.426282] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1696.340243] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1702.534607] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1704.929079] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.932534] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1704.934553] Lustre: Skipped 1 previous similar message [ 1710.307825] Lustre: server umount lustre-MDT0001 complete [ 1713.794121] LustreError: 98453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.705805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1714.769868] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1714.773896] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1714.793147] Lustre: 98445:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1715.107495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1715.110968] Lustre: Skipped 3 previous similar messages [ 1716.713482] LustreError: 102400:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1716.720196] LustreError: 102400:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1716.725262] LustreError: 98448:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1716.771871] LustreError: 102411:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.115@tcp: cfg command failed: rc = -17 [ 1716.779159] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1716.779159] [ 1716.911925] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1720.294852] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1721.522446] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:52:27 (1780325547) [ 1728.885822] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1737.625580] Lustre: 104277:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1748.752446] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1750.338641] Lustre: 105410:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1807.329706] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.329913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1807.334614] Lustre: Skipped 1 previous similar message [ 1807.337480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1807.341979] Lustre: Skipped 9 previous similar messages [ 1812.776425] Lustre: server umount lustre-MDT0000 complete [ 1814.220792] LustreError: 98437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325641 with bad export cookie 7436940097374999309 [ 1814.222250] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1814.230950] LustreError: 98437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1814.349618] Lustre: server umount lustre-MDT0001 complete [ 1825.812895] Lustre: server umount lustre-OST0000 complete [ 1836.556656] Lustre: server umount lustre-OST0001 complete [ 1839.049048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1843.162327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1846.694477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1850.517794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1858.733720] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1863.234421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.249036] Lustre: lustre-MDT0000: reset Object Index mappings [ 1863.442527] LustreError: 109337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1863.483130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.191693] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1868.986468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.001098] Lustre: lustre-MDT0001: reset Object Index mappings [ 1871.207191] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1872.479381] Lustre: 110450:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1875.037390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.701265] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1878.440801] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 1880.919127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1883.528984] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1886.185750] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 1889.264500] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 1889.265125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1893.075499] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1894.526897] Lustre: 112288:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1916.402085] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:55:42 (1780325742) [ 1945.571884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.574961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.584464] Lustre: Skipped 4 previous similar messages [ 1945.598013] Lustre: Skipped 7 previous similar messages [ 1959.391167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1959.474687] Lustre: server umount lustre-MDT0000 complete [ 1960.934804] LustreError: 112304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.941926] LustreError: 112304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1961.108596] LustreError: 110070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780325788 with bad export cookie 7436940097375046090 [ 1961.112469] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1961.113225] LustreError: 110070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1966.049893] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.057246] Lustre: Skipped 1 previous similar message [ 1967.437542] Lustre: server umount lustre-MDT0001 complete [ 1975.263185] Lustre: server umount lustre-OST0000 complete [ 1983.099360] Lustre: server umount lustre-OST0001 complete [ 1986.159380] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 1990.568733] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2007.541721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.670840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2007.693122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2007.753697] Lustre: lustre-MDT0000: new disk, initializing [ 2007.805631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.809912] Lustre: Skipped 3 previous similar messages [ 2007.818782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2010.083567] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2013.923424] Lustre: server umount lustre-MDT0000 complete [ 2018.256429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.401407] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2018.407270] Lustre: 116500:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2018.414731] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2020.125549] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2022.518496] Lustre: server umount lustre-MDT0000 complete [ 2026.797806] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:57:33 (1780325853) [ 2027.505325] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2028.326801] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:57:34 (1780325854) [ 2036.407529] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2041.248296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.220929] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2046.920611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.971039] Lustre: 118511:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2047.001891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2047.004376] Lustre: Skipped 1 previous similar message [ 2047.044089] Lustre: lustre-MDT0001: new disk, initializing [ 2047.099356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2047.103729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2048.683132] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2051.391498] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2054.249931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2054.383539] Lustre: lustre-OST0000: new disk, initializing [ 2054.386085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2054.391447] Lustre: 120058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2054.749513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2054.754350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2054.801749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2056.878453] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2062.661280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2062.736499] Lustre: lustre-OST0001: new disk, initializing [ 2062.738558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2062.742410] Lustre: 120913:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2064.475740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2064.480165] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2064.533133] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2065.394724] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2071.188221] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2073.178439] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2283.251592] Lustre: 127293:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2283.257161] Lustre: 127293:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2283.260623] Lustre: 127293:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2283.264981] Lustre: 127293:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2283.269632] Lustre: 127293:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2283.273466] Lustre: 127293:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2420.264358] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:04:06 (1780326246) [ 2433.508841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2433.517087] Lustre: Skipped 1 previous similar message [ 2433.519210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2433.527451] Lustre: Skipped 13 previous similar messages [ 2438.629571] LustreError: 118520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.640351] LustreError: 118520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2438.711690] Lustre: server umount lustre-MDT0000 complete [ 2440.388127] LustreError: 120057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780326268 with bad export cookie 7436940097375103651 [ 2440.391440] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2440.392929] LustreError: 120057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2446.655124] Lustre: server umount lustre-MDT0001 complete [ 2454.740283] Lustre: server umount lustre-OST0000 complete [ 2462.860544] Lustre: server umount lustre-OST0001 complete [ 2466.168790] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 2471.006633] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2490.453059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.636526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2490.660093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2490.732924] Lustre: lustre-MDT0000: new disk, initializing [ 2490.803174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.805517] Lustre: Skipped 5 previous similar messages [ 2490.815156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2493.379397] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2499.771928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.845892] Lustre: 130284:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2499.990611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2502.007401] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2506.000485] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2507.811280] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2511.429683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2511.619651] Lustre: 132024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2512.942399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2512.954892] Lustre: Skipped 1 previous similar message [ 2512.958146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2513.026765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2514.883423] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2519.458939] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2524.749734] Lustre: lustre-MDD0000: changelog on [ 2526.412913] Lustre: lustre-MDD0001: changelog on [ 3689.627158] Lustre: lustre-MDD0001: changelog off [ 3690.731787] Lustre: 130293:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3691.414526] Lustre: lustre-MDD0000: changelog off [ 3696.097086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3696.102365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.104768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.104775] Lustre: Skipped 4 previous similar messages [ 3696.132217] Lustre: Skipped 4 previous similar messages [ 3701.703808] Lustre: server umount lustre-OST0000 complete [ 3706.337328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.342650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3706.355205] Lustre: Skipped 1 previous similar message [ 3706.359264] Lustre: Skipped 3 previous similar messages [ 3710.459467] Lustre: server umount lustre-MDT0000 complete [ 3711.464709] LustreError: 130296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3711.474502] LustreError: 130296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3713.440545] LustreError: 130276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327541 with bad export cookie 7436940097375868779 [ 3713.445872] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3713.450276] LustreError: 130276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3713.703214] Lustre: server umount lustre-MDT0001 complete [ 3721.170497] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 3722.897538] Key type lgssc unregistered [ 3723.184886] LNet: 140913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3723.199057] LNetError: 140913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3723.230645] LNet: Removed LNI 192.168.202.115@tcp [ 3723.899145] Key type .llcrypt unregistered [ 3723.900456] Key type ._llcrypt unregistered [ 3736.004020] Key type ._llcrypt registered [ 3736.005595] Key type .llcrypt registered [ 3736.067893] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 3745.626106] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3746.241910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3746.311342] alg: No test for adler32 (adler32-zlib) [ 3747.239741] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 3747.380354] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 3749.032489] Key type lgssc registered [ 3749.811613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3773.960390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3773.981311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3775.175315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3775.202987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3775.256357] Lustre: lustre-MDT0000: new disk, initializing [ 3775.299440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3775.309615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3778.230857] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3785.704786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3785.775896] Lustre: 144057:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3785.813403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3785.815790] Lustre: Skipped 1 previous similar message [ 3785.870497] Lustre: lustre-MDT0001: new disk, initializing [ 3785.948225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3785.972550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3785.980880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3788.595147] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3793.209845] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3795.543269] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3799.752595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3799.918492] Lustre: lustre-OST0000: new disk, initializing [ 3799.921095] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3799.924527] Lustre: 145797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3799.981125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3803.582471] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3806.765658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3806.772328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3806.827245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3808.270306] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3811.539788] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_module obdecho/obdecho [ 3812.756861] LustreError: 146669:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3812.762500] LustreError: 146669:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3813.945465] LustreError: 146765:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 3822.049189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3822.051583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.053549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3822.068673] Lustre: Skipped 1 previous similar message [ 3827.088969] Lustre: server umount lustre-MDT0000 complete [ 3829.387142] LustreError: 144834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327657 with bad export cookie 488457945109944182 [ 3829.389921] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3829.395074] LustreError: 144834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3829.611927] Lustre: server umount lustre-MDT0001 complete [ 3842.271962] Lustre: server umount lustre-OST0000 complete [ 3854.119801] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 3860.336355] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3880.625301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3880.767430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3880.789392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3880.838351] Lustre: lustre-MDT0000: new disk, initializing [ 3880.874102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3880.886803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3883.063665] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3889.305179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3889.377691] Lustre: 150442:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3889.408201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3889.410671] Lustre: Skipped 1 previous similar message [ 3889.459922] Lustre: lustre-MDT0001: new disk, initializing [ 3889.522812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3889.552449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3889.559213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3891.694747] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3895.967202] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3897.961303] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3901.908545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3902.109313] Lustre: lustre-OST0000: new disk, initializing [ 3902.112668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3902.115761] Lustre: 152180:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3902.167747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.245992] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3908.104162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3908.108818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3908.148310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3909.666506] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3912.255638] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3912.416693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3915.039569] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3915.147883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3918.305572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.306615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3918.321057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3918.321066] Lustre: Skipped 2 previous similar messages [ 3918.329334] Lustre: Skipped 2 previous similar messages [ 3922.303267] Lustre: server umount lustre-OST0000 complete [ 3925.472617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3925.483250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3925.491453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3925.494991] Lustre: Skipped 1 previous similar message [ 3928.544557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3928.546696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3930.867382] Lustre: server umount lustre-MDT0000 complete [ 3933.098263] LustreError: 150433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327760 with bad export cookie 488457945109945141 [ 3933.103179] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3933.112361] LustreError: 150433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3933.342724] Lustre: server umount lustre-MDT0001 complete [ 3935.592487] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:29:21 (1780327761) [ 3936.709639] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3937.831481] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:29:23 (1780327763) [ 3942.256850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.486448] LustreError: 154443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3942.494189] LustreError: 154443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3942.525348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3944.669075] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3948.000446] LustreError: 154444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.450780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.355255] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3955.760463] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3957.892750] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3962.102250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3966.074260] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3969.783953] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3979.745271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.754335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3979.761709] Lustre: Skipped 2 previous similar messages [ 3979.770314] Lustre: Skipped 3 previous similar messages [ 3980.927857] Lustre: server umount lustre-MDT0000 complete [ 3983.160729] LustreError: 154423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327810 with bad export cookie 488457945109946016 [ 3983.162857] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3983.167995] LustreError: 154423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3984.865215] LustreError: 156054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.896455] LustreError: 156054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.901621] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3988.903948] LustreError: 156054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3988.919832] Lustre: Skipped 3 previous similar messages [ 3989.432784] Lustre: server umount lustre-MDT0001 complete [ 3993.331258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4004.145611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4004.645112] LustreError: 157938:0:(llog.c:1646:llog_backup()) MGC192.168.202.115@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4004.656497] Lustre: 157938:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.115@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4005.062371] LustreError: 157943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.632360] LustreError: 157944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.652308] LustreError: 157944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4037.763761] LustreError: 157944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4037.779892] LustreError: 157944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 4065.759343] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4065.763306] Lustre: 157938:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4072.581857] LustreError: 157943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4072.610306] LustreError: 157943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 4106.306746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4106.315822] Lustre: Skipped 2 previous similar messages [ 4108.324474] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4110.686254] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4119.132582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.882767] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.15@tcp (not set up) [ 4128.047879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 4133.357644] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4133.611265] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4139.833594] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4141.959554] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4146.044590] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4147.700778] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4169.185417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4169.192316] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.195727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4169.221400] Lustre: Skipped 3 previous similar messages [ 4169.989613] Lustre: server umount lustre-MDT0000 complete [ 4174.115054] LustreError: 154424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328001 with bad export cookie 488457945109952470 [ 4174.119415] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4174.125773] LustreError: 154424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4174.304329] LustreError: 157944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4174.329107] LustreError: 157944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 43 previous similar messages [ 4174.541467] Lustre: server umount lustre-MDT0001 complete [ 4181.147948] Lustre: server umount lustre-OST0000 complete [ 4202.175231] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 4210.718626] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4248.608559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4248.883819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4248.907632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4248.971530] Lustre: lustre-MDT0000: new disk, initializing [ 4249.051029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4249.056147] Lustre: Skipped 1 previous similar message [ 4249.073307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4253.851606] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4265.525842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.622916] Lustre: 162786:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4265.905810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4269.932572] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4278.582920] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4282.555301] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4288.560435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4288.766249] Lustre: lustre-OST0000: new disk, initializing [ 4288.769621] Lustre: Skipped 1 previous similar message [ 4288.772928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4288.776418] Lustre: Skipped 2 previous similar messages [ 4288.780817] Lustre: 164527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4293.613582] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4296.264855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4296.269797] Lustre: Skipped 1 previous similar message [ 4296.274276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4296.337281] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4300.728550] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4304.755728] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4304.935347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4309.461841] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4309.624745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4312.044881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4312.052273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4312.056862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4312.056872] Lustre: Skipped 2 previous similar messages [ 4312.064554] Lustre: Skipped 2 previous similar messages [ 4312.089977] LustreError: Skipped 1 previous similar message [ 4317.778754] Lustre: server umount lustre-OST0000 complete [ 4328.358282] Lustre: server umount lustre-MDT0000 complete [ 4331.818151] LustreError: 162777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328159 with bad export cookie 488457945109969298 [ 4331.821942] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4331.826265] LustreError: 162777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4332.237032] Lustre: server umount lustre-MDT0001 complete [ 4341.606889] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:36:07 (1780328167) [ 4348.712248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.068880] LustreError: 166792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.343572] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4363.930701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4369.200726] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4376.982864] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4382.762315] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4395.496668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4396.594652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4396.601257] Lustre: Skipped 4 previous similar messages [ 4405.101095] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4411.796536] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4438.082416] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:37:42 (1780328262) [ 4444.652134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4444.655994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4444.657549] LustreError: Skipped 2 previous similar messages [ 4444.666737] Lustre: Skipped 2 previous similar messages [ 4444.667603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4444.672720] Lustre: Skipped 8 previous similar messages [ 4449.312953] Lustre: server umount lustre-OST0000 complete [ 4469.215149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4469.742494] Lustre: server umount lustre-MDT0000 complete [ 4473.596148] LustreError: 166771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328301 with bad export cookie 488457945109970180 [ 4473.603319] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4474.178893] Lustre: server umount lustre-MDT0001 complete [ 4489.100462] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 4493.018464] Key type lgssc unregistered [ 4493.343898] LNet: 170192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4493.358449] LNetError: 170192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4493.383188] LNet: Removed LNI 192.168.202.115@tcp [ 4494.645578] Key type .llcrypt unregistered [ 4494.654044] Key type ._llcrypt unregistered [ 4514.242148] Key type ._llcrypt registered [ 4514.243669] Key type .llcrypt registered [ 4514.371620] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4515.435419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4515.745065] alg: No test for adler32 (adler32-zlib) [ 4516.872737] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 4517.157513] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 4518.871509] Key type lgssc registered [ 4520.086604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4528.364805] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4528.391079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4544.031397] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4549.151310] LustreError: 171418:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.115@tcp: failed processing log, type 4: rc = -110 [ 4554.271378] LustreError: 171418:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4554.283430] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4554.288860] LustreError: MGC192.168.202.115@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4554.298391] LustreError: 171418:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4554.306551] LustreError: 171418:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 4554.317218] LustreError: 171418:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4554.424581] Lustre: server umount lustre-OST0000 complete [ 4554.426619] LustreError: 171418:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4563.091323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4564.735563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.758366] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4575.219293] LustreError: 171917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4575.255847] LustreError: 171917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4578.398945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.654191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4583.645077] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4588.522342] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4592.749755] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4599.211376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4599.354358] LustreError: 173506:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4599.361292] Lustre: 173506:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.202.115@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4599.483443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4605.990146] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4612.044886] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4614.625451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4614.636929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.663371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4615.153239] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.168666] Lustre: Skipped 1 previous similar message [ 4615.173929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4615.183281] Lustre: Skipped 2 previous similar messages [ 4620.260663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4620.269142] Lustre: Skipped 2 previous similar messages [ 4620.473194] Lustre: server umount lustre-MDT0000 complete [ 4624.002956] LustreError: 171898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328451 with bad export cookie 13612723925072697622 [ 4624.012174] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4624.363785] Lustre: server umount lustre-MDT0001 complete [ 4638.741229] Lustre: server umount lustre-OST0000 complete [ 4648.598912] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 4660.366126] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4701.703376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.026753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4702.064855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4702.168156] Lustre: lustre-MDT0000: new disk, initializing [ 4702.303412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4702.336617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4707.116623] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4721.533882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.694582] Lustre: 176884:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4721.743562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4721.751291] Lustre: Skipped 1 previous similar message [ 4721.858125] Lustre: lustre-MDT0001: new disk, initializing [ 4722.016505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4722.053526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4722.059926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4727.590579] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4737.050932] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4741.496133] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4749.173895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4749.446276] Lustre: lustre-OST0000: new disk, initializing [ 4749.453502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4749.466901] Lustre: 178627:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4749.579644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4756.354595] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4756.603724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4756.619828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4756.774888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4764.923967] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4770.536428] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4770.805653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4776.242471] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4776.441597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4782.567722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4782.569112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.574670] LustreError: Skipped 1 previous similar message [ 4782.579734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4782.601614] Lustre: Skipped 1 previous similar message [ 4784.659535] Lustre: server umount lustre-OST0000 complete [ 4788.706530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4788.714728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4788.730494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4788.737045] Lustre: Skipped 1 previous similar message [ 4792.802564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4797.920391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4797.935878] Lustre: Skipped 3 previous similar messages [ 4802.016769] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4802.338740] Lustre: server umount lustre-MDT0000 complete [ 4803.044671] LustreError: 176891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4803.060246] LustreError: 176891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4806.036671] LustreError: 176875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328633 with bad export cookie 13612723925072698343 [ 4806.044899] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4806.051681] LustreError: 176875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4806.495629] Lustre: server umount lustre-MDT0001 complete [ 4814.948859] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:44:00 (1780328640) [ 4832.623385] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 4835.212335] Key type lgssc unregistered [ 4835.471178] LNet: 181410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4835.477633] LNetError: 181410:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4835.504285] LNet: Removed LNI 192.168.202.115@tcp [ 4836.319648] Key type .llcrypt unregistered [ 4836.322661] Key type ._llcrypt unregistered [ 4852.794457] Key type ._llcrypt registered [ 4852.796431] Key type .llcrypt registered [ 4852.882456] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4853.873995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4854.158459] alg: No test for adler32 (adler32-zlib) [ 4855.182189] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 4855.383143] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 4857.047313] Key type lgssc registered [ 4858.029106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4864.351665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4864.375325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.765790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4869.368412] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4876.257104] LustreError: 182615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4876.271495] LustreError: 182615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4876.439400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4876.684553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4880.858721] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4885.491297] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4889.489601] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4896.413511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4896.536369] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4896.538511] LustreError: 184252:0:(llog.c:1687:llog_backup()) MGC192.168.202.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4896.543649] Lustre: 184252:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.202.115@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4896.630188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4903.275692] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4908.622763] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4911.051466] LustreError: 182635:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4911.068846] LustreError: 182635:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4912.608828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4912.618434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.651498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4916.193544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.194184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4916.238283] Lustre: Skipped 2 previous similar messages [ 4917.519436] Lustre: server umount lustre-MDT0000 complete [ 4921.321489] LustreError: 182611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.365357] LustreError: 182611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4921.950942] LustreError: 182595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328749 with bad export cookie 2608634461552793487 [ 4921.962480] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4921.975108] LustreError: 182595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4922.280660] Lustre: server umount lustre-MDT0001 complete [ 4936.575800] Lustre: server umount lustre-OST0000 complete [ 4946.381141] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 4957.015282] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4991.264862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4991.521995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4991.567125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4991.675485] Lustre: lustre-MDT0000: new disk, initializing [ 4991.771638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4991.794324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4995.990663] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5006.033798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5006.147757] Lustre: 187626:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5006.181692] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5006.183778] Lustre: Skipped 1 previous similar message [ 5006.185894] LustreError: 188393:0:(llog.c:1687:llog_backup()) MGC192.168.202.115@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5006.190621] LustreError: 188393:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 5006.194524] Lustre: 188393:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.115@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5006.206898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5006.210572] Lustre: Skipped 1 previous similar message [ 5006.262432] Lustre: lustre-MDT0001: new disk, initializing [ 5006.334528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5006.356299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5006.362347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5010.225846] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5016.475317] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5020.747969] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5027.013237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5027.204672] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5027.206709] Lustre: Skipped 1 previous similar message [ 5027.209721] LustreError: 189347:0:(llog.c:1687:llog_backup()) MGC192.168.202.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5027.219354] LustreError: 189347:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 5027.223683] Lustre: 189347:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.115@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5027.230616] Lustre: 189347:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5027.246431] Lustre: lustre-OST0000: new disk, initializing [ 5027.249608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5027.254784] Lustre: 189371:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5027.333118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5032.134545] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5033.518216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5033.541574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5033.607634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5038.091111] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5042.071190] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5042.206080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5045.991274] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5046.168563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5049.315267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5049.317897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.319651] LustreError: Skipped 1 previous similar message [ 5049.321130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5049.337946] Lustre: Skipped 2 previous similar messages [ 5054.042552] Lustre: server umount lustre-OST0000 complete [ 5057.504558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5057.515802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.543436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5057.548813] Lustre: Skipped 1 previous similar message [ 5063.791215] Lustre: server umount lustre-MDT0000 complete [ 5064.677390] LustreError: 187636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5064.695100] LustreError: 187636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5066.613386] LustreError: 187620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328894 with bad export cookie 2608634461552794208 [ 5066.614866] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5066.619761] LustreError: 187620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5066.915792] Lustre: server umount lustre-MDT0001 complete [ 5074.220145] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:48:20 (1780328900) [ 5089.220314] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5098.473843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5098.901576] LustreError: 192330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5098.994098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5102.954456] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5104.098262] LustreError: 192331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5110.743526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.202418] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5118.311885] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5124.041730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5129.143468] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5136.903571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5137.156529] Lustre: lustre-OST0001: new disk, initializing [ 5137.160089] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5137.165068] Lustre: 194586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5137.257098] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5137.259872] Lustre: Skipped 2 previous similar messages [ 5142.147638] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5145.653206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5145.659957] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5145.722598] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5151.255804] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5156.338769] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5158.918743] Lustre: DEBUG MARKER: ADD OST3 [ 5162.903953] loop: module loaded [ 5162.955975] loop0: detected capacity change from 0 to 204800000 [ 5163.146447] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5163.152838] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5163.161669] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5168.325647] loop0: detected capacity change from 0 to 204800000 [ 5169.541985] Lustre: DEBUG MARKER: STOP OST3 [ 5173.010313] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5173.260624] Lustre: lustre-OST0003: new disk, initializing [ 5173.265164] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5173.275207] Lustre: 196530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5174.479690] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5174.579898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5174.588811] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5174.643195] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 5176.544663] LustreError: 196695:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5176.713510] Lustre: Failing over lustre-OST0003 [ 5176.962355] Lustre: server umount lustre-OST0003 complete [ 5177.700084] LustreError: 193780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5177.725690] LustreError: 193780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5177.980068] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5179.872220] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5179.885073] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5179.904318] Lustre: Skipped 2 previous similar messages [ 5186.699385] Lustre: DEBUG MARKER: START OST3 again [ 5191.751855] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5191.931952] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5193.352694] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5193.851176] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5193.864741] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5196.419756] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5213.486657] Lustre: Failing over lustre-OST0003 [ 5213.555768] Lustre: server umount lustre-OST0003 complete [ 5214.687914] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5214.696657] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.704736] LustreError: 195622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5214.711018] Lustre: Skipped 1 previous similar message [ 5214.728629] LustreError: 195622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5229.030599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5229.035125] LustreError: Skipped 1 previous similar message [ 5229.039584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5229.044599] Lustre: Skipped 2 previous similar messages [ 5231.078272] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.091465] Lustre: Skipped 3 previous similar messages [ 5234.951708] Lustre: server umount lustre-MDT0000 complete [ 5236.894886] LustreError: 193804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329064 with bad export cookie 2608634461552795090 [ 5236.896262] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5236.906188] LustreError: 193804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5241.316848] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5241.321754] Lustre: Skipped 5 previous similar messages [ 5243.216758] Lustre: server umount lustre-MDT0001 complete [ 5245.340058] Lustre: server umount lustre-OST0000 complete [ 5247.478616] Lustre: server umount lustre-OST0001 complete [ 5250.766716] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 5255.117154] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5276.912880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5277.047268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5277.073701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5277.127759] Lustre: lustre-MDT0000: new disk, initializing [ 5277.180944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5277.184505] Lustre: Skipped 2 previous similar messages [ 5277.198280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5279.560322] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5287.002319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5287.077866] Lustre: 201957:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5287.245116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5290.111986] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5295.531955] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5297.446531] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5301.338531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5301.497948] Lustre: lustre-OST0000: new disk, initializing [ 5301.500192] Lustre: Skipped 1 previous similar message [ 5301.502931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5301.506530] Lustre: Skipped 2 previous similar messages [ 5301.510822] Lustre: 203694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5303.275731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5303.281186] Lustre: Skipped 1 previous similar message [ 5303.293779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5303.357893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5305.012158] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5310.298638] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5313.167417] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5313.294513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5315.966386] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5316.098257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5318.624435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5318.630382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.634725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5318.642539] Lustre: Skipped 3 previous similar messages [ 5323.507847] Lustre: server umount lustre-OST0000 complete [ 5340.128461] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5340.287567] Lustre: server umount lustre-MDT0000 complete [ 5342.762584] LustreError: 201951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329170 with bad export cookie 2608634461552797386 [ 5342.765336] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5342.775900] LustreError: 201951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5342.998567] Lustre: server umount lustre-MDT0001 complete [ 5345.562533] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:52:51 (1780329171) [ 5355.507920] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 5361.732431] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5384.499789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.714877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5384.740767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5384.820584] Lustre: lustre-MDT0000: new disk, initializing [ 5384.915350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5387.626228] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5394.673301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5394.760475] Lustre: 208120:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5394.960615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5397.466475] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5402.221572] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5404.518386] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5408.730131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5408.940132] Lustre: 209860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5408.994344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5408.997463] Lustre: Skipped 4 previous similar messages [ 5410.829431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5412.003445] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5416.541259] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5418.862907] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5418.980555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5420.966727] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5421.084480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5426.153717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.154193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5426.165443] Lustre: Skipped 2 previous similar messages [ 5426.170969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5426.173122] LustreError: Skipped 2 previous similar messages [ 5426.187531] Lustre: Skipped 8 previous similar messages [ 5436.383807] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5436.534415] Lustre: server umount lustre-OST0000 complete [ 5453.279281] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5453.396937] Lustre: server umount lustre-MDT0000 complete [ 5455.368089] LustreError: 208897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329283 with bad export cookie 2608634461552798268 [ 5455.372723] LustreError: 208897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5455.373173] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5455.539947] Lustre: server umount lustre-MDT0001 complete [ 5460.030302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.271253] LustreError: 211930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5460.286185] LustreError: 211930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 5462.726218] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5468.108081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.684457] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5474.095745] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5476.025640] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5479.935429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5483.975578] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5487.341736] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5495.779132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5495.782654] LustreError: Skipped 1 previous similar message [ 5495.789811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5495.795288] Lustre: Skipped 11 previous similar messages [ 5501.657852] Lustre: server umount lustre-OST0000 complete [ 5510.453121] Lustre: server umount lustre-MDT0000 complete [ 5512.603153] LustreError: 211912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329340 with bad export cookie 2608634461552799150 [ 5512.605815] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5512.607034] LustreError: 211912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5512.766721] Lustre: server umount lustre-MDT0001 complete [ 5623.327209] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 5624.975569] Key type lgssc unregistered [ 5625.169983] LNet: 215236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5625.175432] LNetError: 215236:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5625.189943] LNet: Removed LNI 192.168.202.115@tcp [ 5625.698228] Key type .llcrypt unregistered [ 5625.701248] Key type ._llcrypt unregistered [ 5639.880963] Key type ._llcrypt registered [ 5639.883244] Key type .llcrypt registered [ 5639.957254] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 5648.701279] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5649.224795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5649.258758] alg: No test for adler32 (adler32-zlib) [ 5650.164589] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 5650.290698] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 5651.887197] Key type lgssc registered [ 5652.536557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5670.170367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5670.181790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5671.325739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5671.381276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5671.456951] Lustre: lustre-MDT0000: new disk, initializing [ 5671.520877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5671.537297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5673.905091] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5680.841455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.911683] Lustre: 218530:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5680.951429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5680.953446] Lustre: Skipped 1 previous similar message [ 5681.008937] Lustre: lustre-MDT0001: new disk, initializing [ 5681.076757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5681.097059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5681.101477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5683.453232] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5687.730934] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5689.630815] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5693.656447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5693.813584] Lustre: lustre-OST0000: new disk, initializing [ 5693.815479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5693.818360] Lustre: 220268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5693.878071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.421336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5694.433995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5694.501460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5697.057371] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5702.180459] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5704.546641] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5704.703226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5706.746431] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5706.880464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5709.796400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.812281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5709.813168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5709.835078] Lustre: Skipped 1 previous similar message [ 5714.389733] Lustre: server umount lustre-OST0000 complete [ 5716.961599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5716.965557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5716.978504] Lustre: Skipped 1 previous similar message [ 5716.981657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5720.033339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5720.039677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.843617] Lustre: server umount lustre-MDT0000 complete [ 5724.918890] LustreError: 218521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329552 with bad export cookie 6556044695830262660 [ 5724.919777] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5724.922875] LustreError: 218521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5725.097149] Lustre: server umount lustre-MDT0001 complete [ 5734.129129] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5739.520480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.775933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5741.761047] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5745.120607] LustreError: 223036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.131418] LustreError: 223036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5745.481079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.644996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5747.834806] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5749.437657] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5752.650224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5755.492596] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5759.721476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5759.836969] Lustre: lustre-OST0001: new disk, initializing [ 5759.842492] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5759.850645] Lustre: 225288:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5759.904724] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5759.908867] Lustre: Skipped 1 previous similar message [ 5762.643639] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5767.196649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5767.200877] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5767.257996] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5770.669986] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5774.193073] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5779.339728] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:00:05 (1780329605) [ 5781.472255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5781.478205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5781.483437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5781.486074] Lustre: Skipped 1 previous similar message [ 5782.365466] Lustre: server umount lustre-MDT0000 complete [ 5782.498048] LustreError: 223036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5782.505197] LustreError: 223036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5783.756135] LustreError: 223016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329611 with bad export cookie 6556044695830263542 [ 5783.759344] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5783.763940] LustreError: 223016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5783.896889] Lustre: server umount lustre-MDT0001 complete [ 5795.867397] Lustre: server umount lustre-OST0000 complete [ 5807.863447] Lustre: server umount lustre-OST0001 complete [ 5810.581250] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 5814.942840] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5830.646536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.737986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5830.750926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5830.789650] Lustre: lustre-MDT0000: new disk, initializing [ 5830.835581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.846751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5832.543550] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5838.278582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5838.369536] Lustre: 229177:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5838.514062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5841.210395] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5847.671901] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5851.557270] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5859.657854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5860.277588] Lustre: lustre-OST0000: new disk, initializing [ 5860.296853] Lustre: Skipped 1 previous similar message [ 5860.308240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5860.319738] Lustre: Skipped 2 previous similar messages [ 5860.339731] Lustre: 230927:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5861.531703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5861.545056] Lustre: Skipped 1 previous similar message [ 5861.556895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5861.674412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5868.655102] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5886.002810] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5896.894245] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5897.616760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5907.627941] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5907.788102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5913.307626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.320661] Lustre: Skipped 3 previous similar messages [ 5913.486551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5913.501721] Lustre: Skipped 1 previous similar message [ 5923.812898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5923.833680] Lustre: Skipped 3 previous similar messages [ 5926.367507] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5926.675518] Lustre: server umount lustre-OST0000 complete [ 5931.488547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5931.491777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5931.508431] Lustre: Skipped 1 previous similar message [ 5937.306283] Lustre: server umount lustre-MDT0000 complete [ 5939.182096] LustreError: 229979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5939.199885] LustreError: 229979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5942.260884] LustreError: 229168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329769 with bad export cookie 6556044695830265663 [ 5942.295609] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5943.184348] Lustre: server umount lustre-MDT0001 complete [ 5961.399916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5961.880450] LustreError: 233005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.940106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5961.947842] Lustre: Skipped 2 previous similar messages [ 5967.350156] LustreError: 233004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.211053] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5976.545645] LustreError: 233004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.567526] LustreError: 233004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5981.189889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.640252] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 5994.768123] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5998.999058] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6006.442430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6012.961720] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6019.182728] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6166.501165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6166.508560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6166.511590] Lustre: Skipped 1 previous similar message [ 6166.515676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6166.524366] LustreError: Skipped 1 previous similar message [ 6166.542055] Lustre: Skipped 4 previous similar messages [ 6172.007784] Lustre: server umount lustre-OST0000 complete [ 6181.351364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6183.839847] Lustre: server umount lustre-MDT0000 complete [ 6186.978728] LustreError: 234213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6187.003674] LustreError: 234213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6188.553597] LustreError: 232985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330016 with bad export cookie 6556044695830266545 [ 6188.559181] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.575221] LustreError: 232985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6188.905669] Lustre: server umount lustre-MDT0001 complete [ 6294.119327] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 6299.390588] Key type lgssc unregistered [ 6300.067269] LNet: 236259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6300.104745] LNetError: 236259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6300.135432] LNet: Removed LNI 192.168.202.115@tcp [ 6302.247626] Key type .llcrypt unregistered [ 6302.252731] Key type ._llcrypt unregistered [ 6342.765450] Key type ._llcrypt registered [ 6342.767528] Key type .llcrypt registered [ 6342.856430] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 6358.111565] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6359.720746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6359.773616] alg: No test for adler32 (adler32-zlib) [ 6360.829968] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 6361.148542] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 6362.879165] Key type lgssc registered [ 6364.273974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6407.880865] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6417.590744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6417.617761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6418.891590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6418.949973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6419.011272] Lustre: lustre-MDT0000: new disk, initializing [ 6419.086408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6419.102324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6423.188994] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6432.471102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6432.566935] Lustre: 240282:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6432.615736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6432.620869] Lustre: Skipped 1 previous similar message [ 6432.701229] Lustre: lustre-MDT0001: new disk, initializing [ 6432.820648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6432.880384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6432.892177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6437.104117] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6441.784831] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6447.684503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6448.013342] Lustre: lustre-OST0000: new disk, initializing [ 6448.022239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6448.028235] Lustre: 241879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6448.158606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6452.181071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6452.190568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6452.290077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6454.118199] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6463.468704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6463.645773] Lustre: lustre-OST0001: new disk, initializing [ 6463.652028] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6463.661528] Lustre: 242735:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6463.754728] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6469.118678] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6472.279839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6472.289194] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6472.348806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6478.353832] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6482.227957] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6493.159071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6493.162170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6493.169859] Lustre: Skipped 1 previous similar message [ 6494.173476] Lustre: server umount lustre-MDT0000 complete [ 6496.998759] LustreError: 240273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330324 with bad export cookie 968862493834246413 [ 6497.001800] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6497.014110] LustreError: 240273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6497.348747] Lustre: server umount lustre-MDT0001 complete [ 6511.093077] Lustre: server umount lustre-OST0000 complete [ 6524.432781] Lustre: server umount lustre-OST0001 complete [ 6536.599985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6545.371343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6551.983381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6560.489106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6578.470863] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6585.946437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.962345] Lustre: lustre-MDT0000: reset Object Index mappings [ 6586.269169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6589.439648] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6591.458221] LustreError: 247711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6591.478821] LustreError: 247711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6595.557537] LustreError: 247711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6595.635575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6595.656838] Lustre: lustre-MDT0001: reset Object Index mappings [ 6595.925312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6598.987256] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6600.991754] Lustre: 248818:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6605.597539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6610.753956] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6610.977372] LustreError: 249173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6610.989881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6616.045097] LustreError: 249173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.167370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6617.328896] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6617.332805] Lustre: Skipped 1 previous similar message [ 6620.426900] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6621.135424] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6626.556699] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6628.998664] Lustre: 250654:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6631.904523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6631.913990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.920041] Lustre: Skipped 2 previous similar messages [ 6631.923273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6631.927700] Lustre: Skipped 3 previous similar messages [ 6636.003701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6636.007049] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.017016] Lustre: Skipped 1 previous similar message [ 6637.884789] Lustre: server umount lustre-MDT0000 complete [ 6640.360117] LustreError: 247692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330468 with bad export cookie 968862493834253203 [ 6640.373528] LustreError: 247692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6640.376073] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6640.617185] Lustre: server umount lustre-MDT0001 complete [ 6653.726369] Lustre: server umount lustre-OST0000 complete [ 6666.246706] Lustre: server umount lustre-OST0001 complete [ 6676.252142] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:15:02 (1780330502) [ 6685.713897] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 6690.884832] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6710.816669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6710.984415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6711.008103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6711.062174] Lustre: lustre-MDT0000: new disk, initializing [ 6711.111551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6711.124650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6713.150461] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6719.062214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6719.122204] Lustre: 254870:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6719.154749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6719.157373] Lustre: Skipped 1 previous similar message [ 6719.217528] Lustre: lustre-MDT0001: new disk, initializing [ 6719.315887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6719.325637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6722.258952] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6730.747863] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6734.550791] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6743.076902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6743.677417] Lustre: lustre-OST0000: new disk, initializing [ 6743.689654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6743.706440] Lustre: 256611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6745.710672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6745.736801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6745.909451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6749.944521] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6757.902687] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6763.417031] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6763.729863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6770.755125] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6771.153925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6776.814585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6776.818895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.834171] LustreError: Skipped 1 previous similar message [ 6776.842297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6776.861323] Lustre: Skipped 2 previous similar messages [ 6776.895255] Lustre: Skipped 4 previous similar messages [ 6780.202483] Lustre: server umount lustre-OST0000 complete [ 6791.137281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6791.158407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.181441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6794.014383] Lustre: server umount lustre-MDT0000 complete [ 6797.281678] LustreError: 254880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.307543] LustreError: 254880:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6799.860424] LustreError: 254861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330627 with bad export cookie 968862493834255296 [ 6799.868647] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6799.888851] LustreError: 254861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6800.755353] Lustre: server umount lustre-MDT0001 complete [ 6823.827532] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6835.709844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.178821] LustreError: 259382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6836.277496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6836.285934] Lustre: Skipped 2 previous similar messages [ 6841.609246] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6851.691270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6858.493343] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6861.929255] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6869.682890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6877.284622] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6886.967199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6887.291684] Lustre: lustre-OST0001: new disk, initializing [ 6887.304233] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6887.323867] Lustre: 261642:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6888.719853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6888.751253] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6888.921217] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6895.321403] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6906.744169] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6910.424574] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6916.349621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6916.369699] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6923.746574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6923.756616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6923.769044] Lustre: Skipped 1 previous similar message [ 6923.780211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6923.782858] Lustre: Skipped 2 previous similar messages [ 6929.494625] Lustre: server umount lustre-MDT0000 complete [ 6929.899671] LustreError: 259382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.924621] LustreError: 259382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6934.007071] LustreError: 259363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330761 with bad export cookie 968862493834256178 [ 6934.007942] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6934.010952] LustreError: 259363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6934.501064] Lustre: server umount lustre-MDT0001 complete [ 6948.635841] Lustre: server umount lustre-OST0000 complete [ 6950.368679] Lustre: 237808:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780330762/real 1780330762] req@ffff89420a41c380 x1866811507435776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780330778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6950.404221] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.424850] Lustre: Skipped 3 previous similar messages [ 6954.464437] Lustre: 237806:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780330766/real 1780330766] req@ffff8942092ace00 x1866811507436032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780330782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6954.854934] Lustre: server umount lustre-OST0001 complete [ 6978.334544] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6993.101052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6993.536599] LustreError: 264668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.633169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6993.644613] Lustre: Skipped 3 previous similar messages [ 6999.289628] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7010.605583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7016.106650] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7019.123196] Lustre: 265778:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7026.265142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7035.901354] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7039.000529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7046.814830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7052.839681] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7055.401915] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7064.573356] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7069.862154] Lustre: 267622:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7077.549960] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7077.635723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7091.578655] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:21:56 (1780330916) [ 7109.094732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7109.097745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7109.128412] Lustre: Skipped 2 previous similar messages [ 7109.139587] Lustre: Skipped 7 previous similar messages [ 7115.695294] Lustre: server umount lustre-MDT0000 complete [ 7119.338386] LustreError: 265386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7119.379792] LustreError: 265386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 7121.741707] LustreError: 264649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330949 with bad export cookie 968862493834262758 [ 7121.768129] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7136.223385] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7137.043814] Lustre: server umount lustre-MDT0001 complete [ 7147.924613] Lustre: server umount lustre-OST0000 complete [ 7153.186759] Lustre: server umount lustre-OST0001 complete [ 7161.340344] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 7174.612190] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7233.119762] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7246.425664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.945532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7247.009751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7247.143366] Lustre: lustre-MDT0000: new disk, initializing [ 7247.387338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7252.797352] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7265.428651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7265.703414] Lustre: 271838:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7266.152597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7266.166341] Lustre: Skipped 4 previous similar messages [ 7266.221123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7271.670371] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7285.050142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7285.357588] Lustre: 273437:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7287.269895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7294.830605] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7307.538331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7307.941595] Lustre: 274292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7309.920345] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7309.938912] Lustre: Skipped 1 previous similar message [ 7310.089617] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7316.230246] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7326.898569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7331.138304] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7331.142210] Lustre: Skipped 1 previous similar message [ 7349.757586] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:26:15 (1780331175) [ 7353.317523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7353.326374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7353.349568] Lustre: Skipped 3 previous similar messages [ 7353.370597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7353.376963] Lustre: Skipped 10 previous similar messages [ 7355.077490] Lustre: server umount lustre-MDT0000 complete [ 7356.396524] LustreError: 271849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7356.420618] LustreError: 271849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 7359.408310] LustreError: 271830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331187 with bad export cookie 968862493834270437 [ 7359.411174] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7359.420845] LustreError: 271830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7359.989136] Lustre: server umount lustre-MDT0001 complete [ 7375.376657] Lustre: server umount lustre-OST0000 complete [ 7378.144132] Lustre: 237806:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780331189/real 1780331189] req@ffff894209e20700 x1866811507600512/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780331205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7380.546505] Lustre: server umount lustre-OST0001 complete [ 7392.315908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7398.788628] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7409.568502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7417.706261] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7426.119731] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7431.159904] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7438.631652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7446.677913] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7452.032229] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7481.614716] loop1: detected capacity change from 0 to 204800000 [ 7481.913232] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7481.920603] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7481.929537] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7487.281179] loop1: detected capacity change from 0 to 204800000 [ 7493.234235] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7493.305979] LustreError: 276511:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs13721: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7493.325121] LustreError: 276511:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs13721 [ 7493.340848] LustreError: 276511:0:(mgs_handler.c:517:mgs_target_reg()) fs13721-MDT0001: Error checking target index: rc = -13 [ 7493.372744] LustreError: fs13721-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7493.386655] LustreError: 279972:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 7493.398775] LustreError: 279972:0:(tgt_mount.c:2084:server_put_super()) no obd fs13721-MDT0001 [ 7493.404750] LustreError: 279972:0:(tgt_mount.c:116:server_deregister_mount()) fs13721-MDT0001 not registered [ 7493.447981] Lustre: server umount fs13721-MDT0001 complete [ 7493.451634] LustreError: 279972:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7501.400836] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7501.524918] Lustre: Setting parameter fs13721-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13721-MDT0001 [ 7501.585543] Lustre: srv-fs13721-MDT0001: No data found on store. Initialize space. [ 7501.590672] Lustre: Skipped 4 previous similar messages [ 7501.668184] Lustre: fs13721-MDT0001: new disk, initializing [ 7501.673518] Lustre: Skipped 3 previous similar messages [ 7501.835920] Lustre: srv-fs13721-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7501.845071] Lustre: 280438:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13721-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7501.855071] LustreError: 280438:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13721-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7506.475471] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7515.587885] Lustre: server umount fs13721-MDT0001 complete [ 7532.905400] loop1: detected capacity change from 0 to 204800000 [ 7534.588816] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7540.494204] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7541.107206] Lustre: 281992:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13721-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7541.121884] LustreError: 281992:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13721-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7547.560982] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7553.891814] Lustre: server umount fs13721-MDT0001 complete [ 7568.867395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7568.879957] LustreError: Skipped 1 previous similar message [ 7568.888743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7568.916297] Lustre: Skipped 4 previous similar messages [ 7568.936085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7572.540356] Lustre: server umount lustre-MDT0000 complete [ 7583.745695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.948583] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7589.383530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7590.686446] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7600.759785] loop1: detected capacity change from 0 to 204800000 [ 7608.686825] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.200766] Lustre: 284405:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13721-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7609.209612] LustreError: 284405:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13721-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7614.343315] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7614.953402] LustreError: 278130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7614.996525] LustreError: 278130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 103 previous similar messages [ 7620.322398] Lustre: server umount fs13721-MDT0001 complete [ 7644.128865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7649.874047] Lustre: server umount lustre-OST0000 complete [ 7669.216410] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7669.566848] Lustre: server umount lustre-MDT0000 complete [ 7674.026665] LustreError: 276504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331501 with bad export cookie 968862493834273118 [ 7674.029484] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7674.042477] LustreError: 276504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7674.524307] Lustre: server umount lustre-MDT0001 complete [ 7688.634671] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 7692.892987] Key type lgssc unregistered [ 7693.305801] LNet: 286721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7693.325991] LNetError: 286721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7693.347974] LNet: Removed LNI 192.168.202.115@tcp [ 7694.826184] Key type .llcrypt unregistered [ 7694.829329] Key type ._llcrypt unregistered [ 7723.670078] Key type ._llcrypt registered [ 7723.671796] Key type .llcrypt registered [ 7723.795694] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7724.470945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7724.556627] alg: No test for adler32 (adler32-zlib) [ 7725.982866] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 7726.315938] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 7728.096429] Key type lgssc registered [ 7729.209776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7738.982913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7739.024941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.859084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7746.735582] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7751.145441] LustreError: 287981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.170962] LustreError: 287981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7756.259278] LustreError: 287982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7756.263196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7756.958228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7762.222277] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7768.787170] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7773.395561] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7781.395921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7781.871090] LustreError: 289584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7781.942616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7786.057988] LustreError: 289584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7786.100141] LustreError: 289584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7789.517690] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7791.085407] LustreError: 289585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.098858] LustreError: 289585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7796.253847] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7801.326919] LustreError: 289584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7801.350675] LustreError: 289584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7818.386600] LustreError: 289586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7818.418777] LustreError: 289586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7818.461298] loop1: detected capacity change from 0 to 204800000 [ 7818.567410] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7818.579736] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7818.591923] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7824.433372] loop1: detected capacity change from 0 to 204800000 [ 7831.097703] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7831.304436] Lustre: fs204-OST0000: new disk, initializing [ 7831.307817] Lustre: srv-fs204-OST0000: No data found on store. Initialize space. [ 7831.312713] Lustre: 291423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7831.402643] Lustre: fs204-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7838.507712] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7847.169488] Lustre: server umount fs204-OST0000 complete [ 7852.520805] LustreError: 289585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7852.544231] LustreError: 289585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 7863.712258] loop1: detected capacity change from 0 to 204800000 [ 7867.895930] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 7870.346354] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7870.673790] Lustre: fs204-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7877.351371] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7888.462701] loop2: detected capacity change from 0 to 204800000 [ 7888.726108] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7888.738233] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7888.754923] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7895.556767] loop2: detected capacity change from 0 to 204800000 [ 7902.165706] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7902.256040] LustreError: fs204-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7902.267405] LustreError: fs204-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7902.275928] LustreError: 294430:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 7902.287928] LustreError: 294430:0:(tgt_mount.c:2084:server_put_super()) no obd fs204-OST0001 [ 7902.295384] LustreError: 294430:0:(tgt_mount.c:116:server_deregister_mount()) fs204-OST0001 not registered [ 7902.325493] Lustre: server umount fs204-OST0001 complete [ 7902.330146] LustreError: 294430:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7916.136774] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7916.340274] Lustre: fs204-OST0001: new disk, initializing [ 7916.349919] Lustre: srv-fs204-OST0001: No data found on store. Initialize space. [ 7916.359484] Lustre: 294982:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7916.495975] Lustre: fs204-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7919.079501] LustreError: 289585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7919.092503] LustreError: 289585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [ 7923.489980] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7932.697257] Lustre: server umount fs204-OST0000 complete [ 7943.277660] Lustre: server umount fs204-OST0001 complete [ 7956.336366] loop1: detected capacity change from 0 to 204800000 [ 7963.589744] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7963.882343] Lustre: fs204-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7971.137775] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7978.776570] loop2: detected capacity change from 0 to 204800000 [ 7985.434478] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7992.148527] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8001.449320] Lustre: server umount fs204-OST0000 complete [ 8010.824518] Lustre: server umount fs204-OST0001 complete [ 8021.486186] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 8021.494959] LustreError: Skipped 1 previous similar message [ 8027.108360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8027.127336] LustreError: Skipped 1 previous similar message [ 8027.140327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8027.166508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8031.714436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.718064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8031.761338] Lustre: Skipped 1 previous similar message [ 8033.237794] Lustre: server umount lustre-OST0000 complete [ 8038.370690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8038.386276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8038.399890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8041.952424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8041.957024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8043.895386] Lustre: server umount lustre-MDT0000 complete [ 8047.931588] LustreError: 287962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331875 with bad export cookie 5010272971356451753 [ 8047.933587] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8047.955787] LustreError: 287962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8048.712431] Lustre: server umount lustre-MDT0001 complete [ 8061.003435] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 8064.538039] Key type lgssc unregistered [ 8064.986800] LNet: 300841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8065.001873] LNetError: 300841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8065.024823] LNet: Removed LNI 192.168.202.115@tcp [ 8065.978157] Key type .llcrypt unregistered [ 8065.980092] Key type ._llcrypt unregistered [ 8104.012730] Key type ._llcrypt registered [ 8104.018297] Key type .llcrypt registered [ 8104.227294] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 8125.321680] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8126.794444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8126.885701] alg: No test for adler32 (adler32-zlib) [ 8128.610265] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 8129.030817] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 8130.835054] Key type lgssc registered [ 8132.589651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8176.224857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8176.262138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8177.624877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8177.680067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8177.807455] Lustre: lustre-MDT0000: new disk, initializing [ 8177.999946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8178.085743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8183.303238] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8194.084030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8194.280532] Lustre: 304684:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8194.346665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8194.351134] Lustre: Skipped 1 previous similar message [ 8194.439052] Lustre: lustre-MDT0001: new disk, initializing [ 8194.546453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8194.580737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8194.600188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8199.434619] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8207.953696] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8212.933595] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8220.832625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.199861] Lustre: lustre-OST0000: new disk, initializing [ 8221.203934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8221.215594] Lustre: 306426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8221.376933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.066303] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8231.023844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8231.034958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8231.157229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8238.618569] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8241.065447] Lustre: Mounted lustre-client [ 8242.354824] LustreError: 307176:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff894201f2b000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 8242.485449] Lustre: Unmounted lustre-client [ 8256.491152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8256.504695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8256.506077] Lustre: Skipped 1 previous similar message [ 8256.528018] Lustre: Skipped 1 previous similar message [ 8259.525537] Lustre: server umount lustre-OST0000 complete [ 8266.211588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8266.224473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8266.237472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8271.289300] Lustre: server umount lustre-MDT0000 complete [ 8275.744491] LustreError: 304676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332103 with bad export cookie 8397266202423927473 [ 8275.757313] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8275.758568] LustreError: 304676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8276.101639] Lustre: server umount lustre-MDT0001 complete [ 8288.103908] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 8291.450161] Key type lgssc unregistered [ 8298.018686] LNet: 308459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8298.037037] LNetError: 308459:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8373.862531] LNet: Removed LNI 192.168.202.115@tcp [ 8375.141122] Key type .llcrypt unregistered [ 8375.144296] Key type ._llcrypt unregistered [ 8394.139400] Key type ._llcrypt registered [ 8394.147816] Key type .llcrypt registered [ 8394.353325] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 8415.716559] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8417.991244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8418.163459] alg: No test for adler32 (adler32-zlib) [ 8419.741353] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 8420.036672] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 8421.800721] Key type lgssc registered [ 8423.480604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8465.504517] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:44:50 (1780332290) [ 8474.663403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8474.715638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8476.520673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8476.610430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8476.806950] Lustre: lustre-MDT0000: new disk, initializing [ 8477.071676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8477.097249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8482.470096] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8494.475799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.713990] Lustre: 311649:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8494.829188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8494.832996] Lustre: Skipped 1 previous similar message [ 8494.895395] Lustre: lustre-MDT0001: new disk, initializing [ 8494.977795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8495.002943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8495.019351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8499.592602] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8507.762409] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8512.440447] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8520.029766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8520.431420] Lustre: lustre-OST0000: new disk, initializing [ 8520.435949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8520.446176] Lustre: 313393:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8520.565428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8525.919511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8525.942622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8526.117731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8527.306741] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8534.652712] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8546.784793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8546.792968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.798735] LustreError: Skipped 1 previous similar message [ 8546.805930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8546.847495] Lustre: Skipped 1 previous similar message [ 8550.955546] Lustre: server umount lustre-OST0000 complete [ 8556.521342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8556.533295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.553498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8556.563405] Lustre: Skipped 1 previous similar message [ 8562.144765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8562.149292] Lustre: Skipped 3 previous similar messages [ 8567.266662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8567.284464] Lustre: Skipped 1 previous similar message [ 8569.311523] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8569.854753] Lustre: server umount lustre-MDT0000 complete [ 8573.667294] LustreError: 311641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332401 with bad export cookie 16188333790857832209 [ 8573.678625] LustreError: 311641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8573.683098] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8573.992297] Lustre: server umount lustre-MDT0001 complete [ 8587.217427] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 8590.496500] Key type lgssc unregistered [ 8590.878561] LNet: 315023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8590.889694] LNetError: 315023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8590.912638] LNet: Removed LNI 192.168.202.115@tcp [ 8591.948896] Key type .llcrypt unregistered [ 8591.951328] Key type ._llcrypt unregistered [ 8623.989629] Key type ._llcrypt registered [ 8623.993194] Key type .llcrypt registered [ 8624.088649] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8625.555341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8625.637930] alg: No test for adler32 (adler32-zlib) [ 8626.847418] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 8627.152801] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 8628.935933] Key type lgssc registered [ 8630.317222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8639.589847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8639.614116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8641.117843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8646.063653] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8650.723673] LustreError: 316428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8650.739440] LustreError: 316428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8655.744430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.842816] LustreError: 316429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8656.332774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8661.219184] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8668.603437] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8672.753156] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8682.548872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8682.950528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8690.683502] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8698.576270] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8708.024039] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8709.629809] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8715.759894] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8715.786752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8715.796772] Lustre: Skipped 2 previous similar messages [ 8715.815949] Lustre: Skipped 2 previous similar messages [ 8716.997810] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.15@tcp (stopping) [ 8720.006790] Lustre: server umount lustre-MDT0000 complete [ 8720.871143] LustreError: 316428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8720.890175] LustreError: 316428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8724.269217] LustreError: 316410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332552 with bad export cookie 8649309332338850198 [ 8724.275948] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8724.279035] LustreError: 316410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8725.984539] LustreError: 316424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8725.993107] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8726.000290] LustreError: 316424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8726.032968] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8730.017897] LustreError: 316424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8730.019166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8730.045203] LustreError: 316424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8730.070340] Lustre: Skipped 1 previous similar message [ 8730.970471] Lustre: server umount lustre-MDT0001 complete [ 8735.419813] Lustre: server umount lustre-OST0000 complete [ 8744.687983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.469689] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8755.186220] Lustre: 320159:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.115@tcp [ 8757.378112] Lustre: server umount MGS complete [ 8767.491694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8767.994099] LustreError: 320731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8768.065571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8772.728863] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8783.148581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8783.707793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8789.143753] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8798.489964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8799.008630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8805.695670] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8811.202843] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8813.851197] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 8818.531479] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8820.468876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8826.856822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8826.869241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8826.891092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8827.894814] LustreError: 322132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8827.923456] LustreError: 322132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8828.345779] Lustre: server umount lustre-OST0000 complete [ 8832.992900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8833.004957] Lustre: Skipped 2 previous similar messages [ 8838.115638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8838.118034] Lustre: Skipped 2 previous similar messages [ 8847.327197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8849.591473] Lustre: server umount lustre-MDT0000 complete [ 8854.060253] LustreError: 320712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332681 with bad export cookie 8649309332338851045 [ 8854.060931] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8854.073044] LustreError: 320712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8854.439755] Lustre: server umount lustre-MDT0001 complete [ 8868.950863] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 8873.977531] Key type lgssc unregistered [ 8874.363748] LNet: 323910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8874.372975] LNetError: 323910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8874.402689] LNet: Removed LNI 192.168.202.115@tcp [ 8875.495604] Key type .llcrypt unregistered [ 8875.498882] Key type ._llcrypt unregistered [ 8897.204479] Key type ._llcrypt registered [ 8897.209324] Key type .llcrypt registered [ 8897.436431] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 8915.137721] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8916.661491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8917.026269] alg: No test for adler32 (adler32-zlib) [ 8918.401612] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 8918.767582] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 8920.575350] Key type lgssc registered [ 8922.295660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8964.014513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8964.040128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8965.497842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8965.528720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8965.623430] Lustre: lustre-MDT0000: new disk, initializing [ 8965.732635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8965.749551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8972.033952] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8986.094828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8986.277966] Lustre: 327054:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8986.361586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8986.365552] Lustre: Skipped 1 previous similar message [ 8986.486696] Lustre: lustre-MDT0001: new disk, initializing [ 8986.652422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8986.703525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8986.716073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8992.425970] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9001.169275] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9005.609834] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9012.435461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9012.757563] Lustre: lustre-OST0000: new disk, initializing [ 9012.775338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9012.783720] Lustre: 328795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9012.879650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9018.500220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9018.514362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9018.687567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9019.238439] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9027.703778] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9033.287497] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9033.565830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9038.379562] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9038.698611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9044.453986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9044.455314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9044.466936] LustreError: Skipped 1 previous similar message [ 9044.470397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9044.478211] Lustre: Skipped 1 previous similar message [ 9044.522103] Lustre: Skipped 1 previous similar message [ 9047.391257] Lustre: server umount lustre-OST0000 complete [ 9053.154158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9053.164203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9053.188105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9054.689284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9054.689494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9054.728782] Lustre: Skipped 1 previous similar message [ 9058.323262] Lustre: server umount lustre-MDT0000 complete [ 9063.234446] LustreError: 327832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332890 with bad export cookie 1721141005246808926 [ 9063.244363] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9063.250738] LustreError: 327832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9063.721482] Lustre: server umount lustre-MDT0001 complete [ 9073.540499] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:54:59 (1780332899) [ 9091.777469] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 9103.407050] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 9149.042324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9149.332408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9149.377381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9149.481994] Lustre: lustre-MDT0000: new disk, initializing [ 9149.627483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9149.667755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9154.778950] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9166.066453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9166.178694] Lustre: 333378:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9166.264296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9166.272965] Lustre: Skipped 1 previous similar message [ 9166.392764] Lustre: lustre-MDT0001: new disk, initializing [ 9166.476332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9166.516457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9166.535789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9171.754125] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9180.964048] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9185.966806] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9193.296157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9193.582737] Lustre: lustre-OST0000: new disk, initializing [ 9193.587368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9193.594905] Lustre: 335120:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9193.756699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9200.688678] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9203.838950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9203.863617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9204.063835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9208.922220] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9255.393302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9255.396712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9255.407225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9255.422813] Lustre: Skipped 1 previous similar message [ 9255.442874] Lustre: Skipped 1 previous similar message [ 9257.492023] Lustre: server umount lustre-OST0000 complete [ 9264.102222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9264.115086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9264.131174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9268.572667] Lustre: server umount lustre-MDT0000 complete [ 9270.755385] LustreError: 334160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9270.806823] LustreError: 334160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9273.505366] LustreError: 333370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333101 with bad export cookie 1721141005246809808 [ 9273.525606] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9274.142318] Lustre: server umount lustre-MDT0001 complete [ 9290.006308] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9294.183141] Key type lgssc unregistered [ 9294.908303] LNet: 337166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9294.939782] LNetError: 337166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9294.972076] LNet: Removed LNI 192.168.202.115@tcp [ 9296.894559] Key type .llcrypt unregistered [ 9296.902275] Key type ._llcrypt unregistered [ 9408.992958] Key type ._llcrypt registered [ 9409.002163] Key type .llcrypt registered [ 9409.152966] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9409.524727] Key type .llcrypt unregistered [ 9409.526692] Key type ._llcrypt unregistered [ 9500.305632] Key type ._llcrypt registered [ 9500.308989] Key type .llcrypt registered [ 9500.419845] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9500.798286] Key type .llcrypt unregistered [ 9500.802696] Key type ._llcrypt unregistered [ 9532.240266] Key type ._llcrypt registered [ 9532.257521] Key type .llcrypt registered [ 9532.401907] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9532.777452] Key type .llcrypt unregistered [ 9532.782154] Key type ._llcrypt unregistered [ 9572.040449] Key type ._llcrypt registered [ 9572.043737] Key type .llcrypt registered [ 9572.161388] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9572.539329] Key type .llcrypt unregistered [ 9572.541156] Key type ._llcrypt unregistered [ 9673.508479] Key type ._llcrypt registered [ 9673.513595] Key type .llcrypt registered [ 9673.733393] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9674.248675] Key type .llcrypt unregistered [ 9674.250633] Key type ._llcrypt unregistered [ 9705.206860] Key type ._llcrypt registered [ 9705.208727] Key type .llcrypt registered [ 9705.318841] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9705.650904] Key type .llcrypt unregistered [ 9705.653633] Key type ._llcrypt unregistered [ 9739.447277] Key type ._llcrypt registered [ 9739.451086] Key type .llcrypt registered [ 9739.598970] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9740.046386] Key type .llcrypt unregistered [ 9740.059315] Key type ._llcrypt unregistered [ 9772.335362] Key type ._llcrypt registered [ 9772.336666] Key type .llcrypt registered [ 9772.474850] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9772.766299] Key type .llcrypt unregistered [ 9772.767729] Key type ._llcrypt unregistered [ 9810.271098] Key type ._llcrypt registered [ 9810.277555] Key type .llcrypt registered [ 9810.471159] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9810.859378] Key type .llcrypt unregistered [ 9810.861198] Key type ._llcrypt unregistered [ 9849.951517] Key type ._llcrypt registered [ 9849.957261] Key type .llcrypt registered [ 9850.112567] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9850.503523] Key type .llcrypt unregistered [ 9850.520215] Key type ._llcrypt unregistered [ 9920.620497] Key type ._llcrypt registered [ 9920.623778] Key type .llcrypt registered [ 9920.758910] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9921.134429] Key type .llcrypt unregistered [ 9921.135973] Key type ._llcrypt unregistered [ 9952.676991] Key type ._llcrypt registered [ 9952.678666] Key type .llcrypt registered [ 9952.799349] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9953.095452] Key type .llcrypt unregistered [ 9953.100681] Key type ._llcrypt unregistered [ 9989.150161] Key type ._llcrypt registered [ 9989.155104] Key type .llcrypt registered [ 9989.255438] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9989.591301] Key type .llcrypt unregistered [ 9989.598285] Key type ._llcrypt unregistered [10048.638796] Key type ._llcrypt registered [10048.640771] Key type .llcrypt registered [10048.777238] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [10065.251357] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10066.559275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10066.591116] alg: No test for adler32 (adler32-zlib) [10067.641548] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [10067.853989] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [10069.519245] Key type lgssc registered [10070.723743] Lustre: Echo OBD driver; http://www.lustre.org/ [10111.503508] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10111.563809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10112.897644] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10112.956446] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10113.114436] Lustre: tst250fs-MDT0000: new disk, initializing [10113.250396] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10113.268229] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10119.269481] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10132.274089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10132.376184] Lustre: 357298:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10132.424885] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10132.428380] Lustre: Skipped 1 previous similar message [10132.484605] Lustre: tst250fs-MDT0001: new disk, initializing [10132.567694] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10132.637533] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10132.651492] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10137.192098] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10142.310919] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10149.160940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10149.434826] Lustre: tst250fs-OST0000: new disk, initializing [10149.441511] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10149.451792] Lustre: 358896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10149.596054] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10150.026147] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10150.035705] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10150.222600] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10156.154872] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10167.297837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10167.513244] Lustre: tst250fs-OST0001: new disk, initializing [10167.518723] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10167.530118] Lustre: 359752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10167.641918] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10173.676631] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10175.128483] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10175.140581] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10175.186875] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10179.354397] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10182.068305] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [10182.073438] Lustre: Skipped 2 previous similar messages [10204.131653] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10204.142913] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10204.163073] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10206.181918] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10206.184898] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10206.196587] Lustre: Skipped 1 previous similar message [10206.246436] Lustre: Skipped 1 previous similar message [10208.282427] Lustre: server umount tst250fs-MDT0000 complete [10213.171853] LustreError: 358526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334040 with bad export cookie 13941460712158684880 [10213.179085] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10213.189913] LustreError: 358526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10213.702385] Lustre: server umount tst250fs-MDT0001 complete [10228.895812] Lustre: server umount tst250fs-OST0000 complete [10232.802021] Lustre: 355523:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780334044/real 1780334044] req@ffff8943078c1180 x1866815394356736/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780334060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10232.870979] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10232.892749] Lustre: Skipped 1 previous similar message [10233.433685] Lustre: server umount tst250fs-OST0001 complete [10256.870669] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [10268.500175] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10304.140618] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:15:29 (1780334129) [10305.921915] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10325.191562] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [10336.949537] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10375.308458] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10158 sec ========= 13:16:40 (1780334200) [10377.315358] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:16:42 (1780334202) === [10382.195200] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:16:46 (1780334206) === [10427.413857] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [10431.186781] Key type lgssc unregistered [10431.839496] LNet: 368800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10431.849452] LNetError: 368800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10431.897338] LNet: Removed LNI 192.168.202.115@tcp [10433.172167] Key type .llcrypt unregistered [10433.180480] Key type ._llcrypt unregistered