[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508801854 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.003269] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.007976] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010148] LSM: Security Framework initializing [ 0.011065] Yama: becoming mindful. [ 0.012062] SELinux: Initializing. [ 0.013077] *** VALIDATE selinux *** [ 0.022125] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027152] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030151] *** VALIDATE tmpfs *** [ 0.031601] *** VALIDATE proc *** [ 0.032359] *** VALIDATE cgroup *** [ 0.033018] *** VALIDATE cgroup2 *** [ 0.034346] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037036] Spectre V2 : User space: Vulnerable [ 0.038017] Speculative Store Bypass: Vulnerable [ 0.041136] debug: unmapping init [mem 0xffffffff8f859000-0xffffffff8f860fff] [ 0.044191] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045898] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046029] ... version: 2 [ 0.047017] ... bit width: 48 [ 0.048020] ... generic registers: 4 [ 0.049017] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.053384] rcu: Hierarchical SRCU implementation. [ 0.055564] smp: Bringing up secondary CPUs ... [ 0.056640] x86: Booting SMP configuration: [ 0.057035] .... node #0, CPUs: #1 #2 #3 [ 0.060381] smp: Brought up 1 node, 4 CPUs [ 0.062018] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105642] node 0 deferred pages initialised in 40ms [ 0.110220] devtmpfs: initialized [ 0.111316] x86/mm: Memory block size: 128MB [ 0.114101] gcov: version magic: 0x41383552 [ 0.116440] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.121100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123612] pinctrl core: initialized pinctrl subsystem [ 0.126248] [ 0.126933] ************************************************************* [ 0.130019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.132012] ** ** [ 0.135031] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.138020] ** ** [ 0.141017] ** This means that this kernel is built to expose internal ** [ 0.144017] ** IOMMU data structures, which may compromise security on ** [ 0.146024] ** your system. ** [ 0.149012] ** ** [ 0.152020] ** If you see this message and you are not debugging the ** [ 0.154012] ** kernel, report this immediately to your vendor! ** [ 0.157013] ** ** [ 0.160045] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163016] ************************************************************* [ 0.166000] NET: Registered protocol family 16 [ 0.169577] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179110] cpuidle: using governor menu [ 0.181000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183587] PCI: Using configuration type 1 for base access [ 0.186129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198078] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.205078] cryptd: max_cpu_qlen set to 1000 [ 0.208211] ACPI: Added _OSI(Module Device) [ 0.210026] ACPI: Added _OSI(Processor Device) [ 0.211019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218083] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225568] ACPI: Interpreter enabled [ 0.227065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229017] ACPI: Using IOAPIC for interrupt routing [ 0.231131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235420] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246860] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.251018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258452] acpiphp: Slot [2] registered [ 0.260177] acpiphp: Slot [5] registered [ 0.262175] acpiphp: Slot [6] registered [ 0.264165] acpiphp: Slot [7] registered [ 0.266173] acpiphp: Slot [8] registered [ 0.268137] acpiphp: Slot [9] registered [ 0.269113] acpiphp: Slot [10] registered [ 0.271152] acpiphp: Slot [3] registered [ 0.273146] acpiphp: Slot [4] registered [ 0.274103] acpiphp: Slot [11] registered [ 0.276127] acpiphp: Slot [12] registered [ 0.278143] acpiphp: Slot [13] registered [ 0.279138] acpiphp: Slot [14] registered [ 0.280121] acpiphp: Slot [15] registered [ 0.281175] acpiphp: Slot [16] registered [ 0.283121] acpiphp: Slot [17] registered [ 0.284172] acpiphp: Slot [18] registered [ 0.286126] acpiphp: Slot [19] registered [ 0.287191] acpiphp: Slot [20] registered [ 0.289102] acpiphp: Slot [21] registered [ 0.290240] acpiphp: Slot [22] registered [ 0.292159] acpiphp: Slot [23] registered [ 0.293169] acpiphp: Slot [24] registered [ 0.295105] acpiphp: Slot [25] registered [ 0.296101] acpiphp: Slot [26] registered [ 0.298161] acpiphp: Slot [27] registered [ 0.299128] acpiphp: Slot [28] registered [ 0.301122] acpiphp: Slot [29] registered [ 0.302123] acpiphp: Slot [30] registered [ 0.304123] acpiphp: Slot [31] registered [ 0.305067] PCI host bridge to bus 0000:00 [ 0.306034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.315034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.318026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.321033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324231] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.327155] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330588] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340857] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353037] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.356647] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359994] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366453] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.384906] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.388885] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.395742] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.404020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.410021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.426035] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.438111] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.488024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.503680] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.509023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.515021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.532027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.546569] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.559029] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.568023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.583026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.594099] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.600020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.609000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.649024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.670094] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.679000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.692029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.710023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.723621] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.726520] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.728386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.732761] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.735641] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.740062] iommu: Default domain type: Passthrough [ 0.741515] SCSI subsystem initialized [ 0.744328] ACPI: bus type USB registered [ 0.745164] usbcore: registered new interface driver usbfs [ 0.748163] usbcore: registered new interface driver hub [ 0.750126] usbcore: registered new device driver usb [ 0.752242] pps_core: LinuxPPS API ver. 1 registered [ 0.754022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.757107] PTP clock support registered [ 0.761167] EDAC MC: Ver: 3.0.0 [ 0.763363] PCI: Using ACPI for IRQ routing [ 0.766324] NetLabel: Initializing [ 0.768017] NetLabel: domain hash size = 128 [ 0.769013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.772106] NetLabel: unlabeled traffic allowed by default [ 0.775056] vgaarb: loaded [ 0.776396] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.778021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.784659] clocksource: Switched to clocksource kvm-clock [ 0.907842] VFS: Disk quotas dquot_6.6.0 [ 0.909664] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.912677] *** VALIDATE ramfs *** [ 0.914071] *** VALIDATE hugetlbfs *** [ 0.916570] pnp: PnP ACPI init [ 0.919335] pnp: PnP ACPI: found 6 devices [ 0.943806] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.948541] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.951696] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.956229] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.959062] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.961730] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.964877] NET: Registered protocol family 2 [ 0.967642] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.973170] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.977292] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.982887] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.987153] TCP: Hash tables configured (established 65536 bind 65536) [ 0.991337] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.995275] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.998769] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.002621] NET: Registered protocol family 1 [ 1.005751] RPC: Registered named UNIX socket transport module. [ 1.008715] RPC: Registered udp transport module. [ 1.011832] RPC: Registered tcp transport module. [ 1.013610] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.015837] NET: Registered protocol family 44 [ 1.017890] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.020978] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.023911] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.027872] PCI: CLS 0 bytes, default 64 [ 1.030559] Unpacking initramfs... [ 2.516111] debug: unmapping init [mem 0xffff962d7cc54000-0xffff962d7ffbffff] [ 2.520050] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.523351] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.526642] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.051613] Initialise system trusted keyrings [ 3.055542] Key type blacklist registered [ 3.059304] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.069460] zbud: loaded [ 3.073311] *** VALIDATE nfs *** [ 3.074586] *** VALIDATE nfs4 *** [ 3.076106] pstore: using deflate compression [ 3.079099] Platform Keyring initialized [ 3.190267] NET: Registered protocol family 38 [ 3.192117] Key type asymmetric registered [ 3.193726] Asymmetric key parser 'x509' registered [ 3.195354] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.198566] io scheduler mq-deadline registered [ 3.200285] io scheduler kyber registered [ 3.202064] io scheduler bfq registered [ 3.204111] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.207531] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.210589] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.213660] ACPI: Power Button [PWRF] [ 3.219443] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.227550] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.242491] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.250092] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.271877] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.302107] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.331628] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.340335] Non-volatile memory driver v1.3 [ 3.342292] Linux agpgart interface v0.103 [ 3.383352] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.387152] vda: detected capacity change from 0 to 76775424 [ 3.406095] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.410429] vdb: detected capacity change from 0 to 1073741824 [ 3.427488] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.430635] vdc: detected capacity change from 0 to 2621440000 [ 3.450160] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.453267] vdd: detected capacity change from 0 to 2621440000 [ 3.470640] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.473727] vde: detected capacity change from 0 to 4294967296 [ 3.491213] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.494401] vdf: detected capacity change from 0 to 4294967296 [ 3.505253] libphy: Fixed MDIO Bus: probed [ 3.513948] usbcore: registered new interface driver usbserial_generic [ 3.517259] usbserial: USB Serial support registered for generic [ 3.520216] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.525895] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.528041] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.530529] mousedev: PS/2 mouse device common for all mice [ 3.533772] rtc_cmos 00:05: RTC can wake from S4 [ 3.537155] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.538693] rtc_cmos 00:05: registered as rtc0 [ 3.547267] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.550038] intel_pstate: CPU model not supported [ 3.551149] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.556255] hid: raw HID events driver (C) Jiri Kosina [ 3.556321] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.558217] usbcore: registered new interface driver usbhid [ 3.563485] usbhid: USB HID core driver [ 3.565233] drop_monitor: Initializing network drop monitor service [ 3.567517] Initializing XFRM netlink socket [ 3.569449] NET: Registered protocol family 10 [ 3.572973] Segment Routing with IPv6 [ 3.574668] NET: Registered protocol family 17 [ 3.577636] mpls_gso: MPLS GSO support [ 3.583639] RAS: Correctable Errors collector initialized. [ 3.585842] AVX version of gcm_enc/dec engaged. [ 3.587601] AES CTR mode by8 optimization enabled [ 3.675477] sched_clock: Marking stable (3675448008, 0)->(4660526710, -985078702) [ 3.678699] registered taskstats version 1 [ 3.680428] Loading compiled-in X.509 certificates [ 3.682138] zswap: loaded using pool lzo/zbud [ 3.708098] Key type big_key registered [ 3.722083] Key type encrypted registered [ 3.723746] ima: No TPM chip found, activating TPM-bypass! [ 3.726718] ima: Allocated hash algorithm: sha1 [ 3.729107] ima: No architecture policies found [ 3.731126] evm: Initialising EVM extended attributes: [ 3.733297] evm: security.selinux [ 3.734644] evm: security.ima [ 3.735976] evm: security.capability [ 3.737251] evm: HMAC attrs: 0x1 [ 3.740153] rtc_cmos 00:05: setting system clock to 2026-09-10 17:56:53 UTC (1789063013) [ 3.747265] debug: unmapping init [mem 0xffffffff90803000-0xffffffff909fffff] [ 3.751371] debug: unmapping init [mem 0xffffffff8f582000-0xffffffff8f858fff] [ 3.763123] Write protecting the kernel read-only data: 28672k [ 3.769344] debug: unmapping init [mem 0xffffffff8dc03000-0xffffffff8ddfffff] [ 3.772452] debug: unmapping init [mem 0xffffffff8e514000-0xffffffff8e5fffff] [ 3.812581] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.823816] systemd[1]: Detected virtualization kvm. [ 3.825970] systemd[1]: Detected architecture x86-64. [ 3.828287] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.852427] systemd[1]: No hostname configured. [ 3.853738] systemd[1]: Set hostname to . [ 3.855553] random: systemd: uninitialized urandom read (16 bytes read) [ 3.858238] systemd[1]: Initializing machine ID from random generator. [ 3.908255] random: ln: uninitialized urandom read (6 bytes read) [ 4.002661] random: systemd: uninitialized urandom read (16 bytes read) [ 4.005766] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.011460] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.019389] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.686865] device-mapper: uevent: version 1.0.3 [ 4.689608] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 5.355015] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.413576] virtio_net virtio0 ens2: renamed from eth0 [ 5.594172] scsi host0: ata_piix [ 5.617376] scsi host1: ata_piix [ 5.661356] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.664172] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.274161] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.242439] random: crng init done [ 10.243841] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.776705] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.938291] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.244433] SELinux: Disabled at runtime. [ 12.309157] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.319681] systemd[1]: Detected virtualization kvm. [ 12.321495] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.874949] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.878280] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.883028] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.887568] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.891211] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.899588] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.910250] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ 12.982946] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.358318] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.676291] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.718210] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.846558] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.864251] EDAC sbridge: Ver: 1.1.2 [ 15.864330] Key type dns_resolver registered [ 16.186548] NFS: Registering the id_resolver key type [ 16.189666] Key type id_resolver registered [ 16.192100] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 51.457573] libcfs: loading out-of-tree module taints kernel. [ 51.525879] Key type ._llcrypt registered [ 51.530443] Key type .llcrypt registered [ 51.673858] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 70.127623] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 71.704945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 71.715633] alg: No test for adler32 (adler32-zlib) [ 73.464607] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 74.863731] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 75.067905] hrtimer: interrupt took 3890123 ns [ 76.655223] Key type lgssc registered [ 78.797980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.307040] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.294632] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 181.947199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 182.015656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.443433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 183.576893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 183.699716] Lustre: lustre-MDT0000: new disk, initializing [ 183.818336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.834890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.648699] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.614026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.743685] Lustre: 6530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 205.783367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 205.785866] Lustre: Skipped 1 previous similar message [ 205.877766] Lustre: lustre-MDT0001: new disk, initializing [ 205.960624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.986292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 205.998941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 211.704909] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.703378] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 230.594690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.871833] Lustre: lustre-OST0000: new disk, initializing [ 230.875929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 230.887597] Lustre: 8469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 230.975032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.775094] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.677644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.694605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.792278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 255.982455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.189612] Lustre: lustre-OST0001: new disk, initializing [ 256.194869] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 256.201776] Lustre: 9554:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.293864] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 261.711412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.721067] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.840936] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 264.008418] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.072588] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 285.563877] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 294.046204] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 301.277708] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 307.821600] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 311.306437] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 314.403420] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 316.845415] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 14:02:03 EDT 2026 [ 336.982305] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 338.565765] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.392947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.403501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.429738] Lustre: Skipped 1 previous similar message [ 359.446348] Lustre: Skipped 3 previous similar messages [ 361.899819] Lustre: server umount lustre-MDT0000 complete [ 367.017114] LustreError: 9564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063376 with bad export cookie 2241965322783580309 [ 367.017854] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 367.029329] LustreError: 9564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 369.636092] LustreError: 11446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.637193] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.639545] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.670682] LustreError: 11446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 369.706630] Lustre: Skipped 3 previous similar messages [ 372.642026] LustreError: 6537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.642449] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.650867] LustreError: 6537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 372.654181] Lustre: Skipped 2 previous similar messages [ 373.344097] Lustre: server umount lustre-MDT0001 complete [ 377.491591] Lustre: server umount lustre-OST0000 complete [ 382.591029] Lustre: server umount lustre-OST0001 complete [ 392.818397] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 404.705468] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 449.515949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.870665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 449.915989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 450.071740] Lustre: lustre-MDT0000: new disk, initializing [ 450.215949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.264877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 456.637825] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 470.290779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.431698] Lustre: 15111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 470.493547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 470.498369] Lustre: Skipped 1 previous similar message [ 470.555367] Lustre: lustre-MDT0001: new disk, initializing [ 470.648297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 470.711236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 470.725733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 476.561368] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 489.374155] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 496.342577] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 505.631874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 506.067909] Lustre: lustre-OST0000: new disk, initializing [ 506.075460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 506.097160] Lustre: 17083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 506.292585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.533292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 507.543366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 507.813285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 513.976736] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 529.279731] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 535.269875] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 535.627567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 542.958904] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 543.232857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 548.836901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 548.844774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.851645] LustreError: Skipped 1 previous similar message [ 548.855564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 548.874664] Lustre: Skipped 1 previous similar message [ 548.917771] Lustre: Skipped 1 previous similar message [ 552.046843] Lustre: server umount lustre-OST0000 complete [ 559.074563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.095738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.109029] Lustre: Skipped 1 previous similar message [ 568.290541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 568.293110] Lustre: Skipped 2 previous similar messages [ 572.387516] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 572.640846] Lustre: server umount lustre-MDT0000 complete [ 573.418069] LustreError: 15119:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.438078] LustreError: 15119:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 577.174527] LustreError: 15103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063586 with bad export cookie 2241965322783581842 [ 577.176953] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 577.191671] LustreError: 15103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 577.630723] Lustre: server umount lustre-MDT0001 complete [ 584.164602] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:06:31 (1789063591) [ 607.984741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.608074] LustreError: 19962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.680672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.859405] LustreError: 19963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.305578] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 624.100154] LustreError: 19963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.118723] LustreError: 19963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 624.422562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.831231] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 640.898970] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 648.098415] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 650.762887] Lustre: Failing over lustre-MDT0000 [ 651.283607] Lustre: server umount lustre-MDT0000 complete [ 654.822805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.829486] LustreError: 21204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.838620] Lustre: Skipped 2 previous similar messages [ 671.071087] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789063664/real 1789063664] req@ffff962cc3770e00 x1875968612540544/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789063680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 671.094170] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 671.970490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.450763] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1f1d0f9527426c18 [ 680.483192] Lustre: MGC192.168.204.146@tcp: Connection restored to 0@lo (at 0@lo) [ 681.074153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.097135] Lustre: Skipped 1 previous similar message [ 681.263421] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 686.057535] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 686.067277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 686.090684] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 689.114800] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 703.665156] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 711.650862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.661936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 711.681445] Lustre: Skipped 1 previous similar message [ 711.681987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 711.688312] Lustre: Skipped 1 previous similar message [ 722.399147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 722.669410] Lustre: server umount lustre-MDT0000 complete [ 724.964055] LustreError: 19957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 724.987657] LustreError: 19957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 726.664917] LustreError: 19942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063736 with bad export cookie 2241965322783583256 [ 726.689144] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 726.691035] LustreError: 19942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 727.280944] Lustre: server umount lustre-MDT0001 complete [ 737.382570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.500640] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 753.739733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.373879] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 768.684981] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 776.502393] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 778.790775] Lustre: Failing over lustre-MDT0000 [ 779.322063] Lustre: server umount lustre-MDT0000 complete [ 779.746947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 779.755513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 794.080318] LustreError: 23310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 794.111556] LustreError: 23310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 799.774588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.971274] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 800.350449] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 805.348604] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 805.359785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 805.378964] Lustre: Skipped 1 previous similar message [ 805.446097] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 805.933473] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 817.072741] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 823.457775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.461807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 823.488423] Lustre: Skipped 2 previous similar messages [ 823.523742] Lustre: Skipped 5 previous similar messages [ 827.789560] Lustre: server umount lustre-MDT0000 complete [ 832.016702] LustreError: 23290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063841 with bad export cookie 2241965322783584194 [ 832.027946] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 832.035602] LustreError: 23290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 832.527750] Lustre: server umount lustre-MDT0001 complete [ 844.459959] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:10:51 (1789063851) [ 865.936154] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 881.828204] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 939.116475] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 954.435888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.679708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 954.722795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 954.814178] Lustre: lustre-MDT0000: new disk, initializing [ 954.950393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.959740] Lustre: Skipped 3 previous similar messages [ 954.974156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 960.148149] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 972.898774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.033940] Lustre: 29734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 973.365145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 979.119787] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 991.499524] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 999.150275] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1008.395309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1008.682991] Lustre: 31705:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1010.701511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1010.755265] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1010.862578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1010.878380] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1016.717585] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1028.512749] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1035.125689] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1061.860249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1061.875710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.900171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1063.986569] Lustre: server umount lustre-OST0000 complete [ 1070.567319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1070.589567] LustreError: Skipped 1 previous similar message [ 1083.359951] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1083.779645] Lustre: server umount lustre-MDT0000 complete [ 1087.972683] LustreError: 29747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1087.987394] LustreError: 29747:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 1088.538108] LustreError: 29728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064098 with bad export cookie 2241965322783584600 [ 1088.544454] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1088.573727] LustreError: 29728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1089.304748] Lustre: server umount lustre-MDT0001 complete [ 1104.342359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1107.830230] Key type lgssc unregistered [ 1108.235386] LNet: 33682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.242833] LNetError: 33682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.260582] LNet: Removed LNI 192.168.204.146@tcp [ 1109.328212] Key type .llcrypt unregistered [ 1109.330471] Key type ._llcrypt unregistered [ 1144.321618] Key type ._llcrypt registered [ 1144.324362] Key type .llcrypt registered [ 1144.489042] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1145.599982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1145.691330] alg: No test for adler32 (adler32-zlib) [ 1146.740637] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1146.915131] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1148.567299] Key type lgssc registered [ 1149.650024] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1159.620568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1159.655605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.455467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1168.797107] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1171.939129] LustreError: 35103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.989852] LustreError: 35103:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1177.059225] LustreError: 35104:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.180458] LustreError: 35103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.042144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1183.364318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1188.531548] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1200.449421] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1208.213739] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1220.553276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1221.509804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.745984] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1229.346134] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1231.844569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1241.646249] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1262.118903] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:17:49 (1789064269) [ 1265.293627] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1267.739349] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1278.342262] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:18:05 (1789064285) [ 1290.468359] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:18:17 (1789064297) [ 1291.850486] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1302.576635] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1302.581919] Lustre: Skipped 2 previous similar messages [ 1305.782332] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:18:32 (1789064312) [ 1308.060365] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1308.067020] Lustre: Skipped 2 previous similar messages [ 1316.366377] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1316.375041] Lustre: Skipped 1 previous similar message [ 1334.404509] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1334.414520] Lustre: Skipped 5 previous similar messages [ 1338.362360] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:19:05 (1789064345) [ 1341.796835] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1346.877698] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1357.557936] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:19:25 (1789064365) [ 1373.840512] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1373.853085] Lustre: Skipped 7 previous similar messages [ 1376.277892] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:19:43 (1789064383) [ 1397.737046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1397.748198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.765934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1400.810158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1400.811415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.828452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1402.343741] Lustre: server umount lustre-OST0000 complete [ 1408.480373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1408.491640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1408.501275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1408.503671] Lustre: Skipped 1 previous similar message [ 1411.041262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1411.054298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.591594] Lustre: server umount lustre-MDT0000 complete [ 1416.163520] LustreError: 35104:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1416.195326] LustreError: 35104:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1418.489274] LustreError: 39644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064428 with bad export cookie 4035704014649156404 [ 1418.500338] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1418.502757] LustreError: 39644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1419.023855] Lustre: server umount lustre-MDT0001 complete [ 1432.977411] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1436.220151] Key type lgssc unregistered [ 1436.466319] LNet: 42974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.495551] LNetError: 42974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.531884] LNet: Removed LNI 192.168.204.146@tcp [ 1437.676665] Key type .llcrypt unregistered [ 1437.681724] Key type ._llcrypt unregistered [ 1459.964907] Key type ._llcrypt registered [ 1459.969806] Key type .llcrypt registered [ 1460.110186] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 1482.404602] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1484.497240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1484.542700] alg: No test for adler32 (adler32-zlib) [ 1485.920133] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1486.132341] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1487.823224] Key type lgssc registered [ 1489.273920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1533.530327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1533.573783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.909833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1534.944654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1535.048320] Lustre: lustre-MDT0000: new disk, initializing [ 1535.153975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.180208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1542.095521] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1554.934362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1555.081557] Lustre: 46129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1555.175176] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1555.183494] Lustre: Skipped 1 previous similar message [ 1555.288929] Lustre: lustre-MDT0001: new disk, initializing [ 1555.380067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1555.424273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1555.431597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1560.513489] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1573.639964] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1580.526793] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1589.675329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1589.991581] Lustre: lustre-OST0000: new disk, initializing [ 1589.995396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1590.006699] Lustre: 48097:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1590.095775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1597.438526] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1599.582856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1599.590223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1599.749793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1608.452104] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1612.852397] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1613.108519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1619.084786] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1619.300106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1625.574459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1625.578084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.601612] LustreError: Skipped 1 previous similar message [ 1625.603611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1625.647814] Lustre: Skipped 1 previous similar message [ 1627.691378] Lustre: server umount lustre-OST0000 complete [ 1632.233179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1632.247484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.276446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.280335] Lustre: Skipped 1 previous similar message [ 1635.812150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.816519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1635.836827] Lustre: Skipped 1 previous similar message [ 1638.119344] Lustre: server umount lustre-MDT0000 complete [ 1642.553826] LustreError: 46121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064652 with bad export cookie 2438313873895763553 [ 1642.567898] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1642.572834] LustreError: 46121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1642.921953] Lustre: server umount lustre-MDT0001 complete [ 1653.610983] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:24:21 (1789064661) [ 1661.690061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.524840] LustreError: 50558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.552496] LustreError: 50558:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1662.646815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.859717] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1668.066497] LustreError: 50559:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1673.192310] LustreError: 50558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.309504] LustreError: 50559:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.676280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.996057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1684.802214] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1694.090429] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1701.275236] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1710.620699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1710.962972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.511524] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1727.699528] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1737.365314] LustreError: 53090:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1738.383165] Lustre: Modifying parameter general.timeout=1 in log params [ 1739.444651] Lustre: Modifying parameter general.timeout=2 in log params [ 1742.179257] Lustre: Modifying parameter general.timeout=5 in log params [ 1742.184695] Lustre: Skipped 2 previous similar messages [ 1746.642460] Lustre: Modifying parameter general.timeout=9 in log params [ 1746.650237] Lustre: Skipped 3 previous similar messages [ 1755.553946] Lustre: Modifying parameter general.timeout=18 in log params [ 1755.564110] Lustre: Skipped 8 previous similar messages [ 1772.949804] Lustre: Modifying parameter general.timeout=34 in log params [ 1772.962719] Lustre: Skipped 15 previous similar messages [ 1806.389434] Lustre: Modifying parameter general.timeout=61 in log params [ 1806.393791] Lustre: Skipped 26 previous similar messages [ 1871.471536] Lustre: Modifying parameter general.timeout=119 in log params [ 1871.476922] Lustre: Skipped 57 previous similar messages [ 1895.498454] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:28:23 (1789064903) [ 1899.184974] Lustre: 60048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1901.382629] Lustre: 60156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1903.545079] Lustre: 60254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1908.137358] Lustre: 60451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1908.146842] Lustre: 60451:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1942.124411] Lustre: 61927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1942.142600] Lustre: 61927:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1942.171602] LustreError: 61927:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1947.096978] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:29:14 (1789064954) [ 1964.002648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.007901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.022817] Lustre: Skipped 2 previous similar messages [ 1964.039684] Lustre: Skipped 2 previous similar messages [ 1967.292677] Lustre: server umount lustre-MDT0000 complete [ 1969.120980] LustreError: 52105:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.147061] LustreError: 52105:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1971.684902] LustreError: 53982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064981 with bad export cookie 2438313873895764435 [ 1971.696864] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1971.700066] LustreError: 53982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1972.142755] Lustre: server umount lustre-MDT0001 complete [ 1985.972304] Lustre: server umount lustre-OST0000 complete [ 2018.977246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.319251] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2019.363926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2019.373944] Lustre: Skipped 33 previous similar messages [ 2019.675206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2024.087828] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2034.072132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.167380] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2034.214053] Lustre: 64460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2038.823947] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2049.292807] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2056.878083] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2067.412806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2067.690036] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2068.217280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.233876] Lustre: Skipped 1 previous similar message [ 2078.054995] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2087.383609] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2090.057127] Lustre: 67028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2098.656640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2098.665042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.674347] LustreError: Skipped 1 previous similar message [ 2098.677409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2098.686451] Lustre: Skipped 1 previous similar message [ 2098.711556] Lustre: Skipped 1 previous similar message [ 2104.825262] Lustre: server umount lustre-OST0000 complete [ 2108.900804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.905971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.928216] Lustre: Skipped 1 previous similar message [ 2108.935452] Lustre: Skipped 3 previous similar messages [ 2115.066810] Lustre: server umount lustre-MDT0000 complete [ 2119.138185] LustreError: 64467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.176319] LustreError: 64467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2120.174349] LustreError: 64454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065129 with bad export cookie 2438313873895767725 [ 2120.191656] LustreError: 64454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2120.193090] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2120.585127] Lustre: server umount lustre-MDT0001 complete [ 2134.961727] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2137.955821] Key type lgssc unregistered [ 2138.298099] LNet: 68242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.311378] LNetError: 68242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.331280] LNet: Removed LNI 192.168.204.146@tcp [ 2139.142760] Key type .llcrypt unregistered [ 2139.145101] Key type ._llcrypt unregistered [ 2170.497519] Key type ._llcrypt registered [ 2170.499824] Key type .llcrypt registered [ 2170.595731] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2172.011082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2172.100060] alg: No test for adler32 (adler32-zlib) [ 2173.388342] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2173.732076] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2175.455244] Key type lgssc registered [ 2176.767692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2186.262821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2186.297512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.728451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.698135] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2197.986447] LustreError: 69654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.019451] LustreError: 69654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2200.593961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.336142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2206.500792] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2215.239616] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2221.513713] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2229.693186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2230.344487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.711185] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2248.141594] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2256.061410] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2266.601833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.614112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2266.627432] Lustre: Skipped 2 previous similar messages [ 2266.640538] Lustre: Skipped 2 previous similar messages [ 2271.726248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.836775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.852554] Lustre: Skipped 4 previous similar messages [ 2278.367185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2278.579376] Lustre: server umount lustre-MDT0000 complete [ 2281.960800] LustreError: 70395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.976171] LustreError: 70395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2283.005739] LustreError: 69635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065292 with bad export cookie 10811608639885541543 [ 2283.009745] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2283.011180] LustreError: 69635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2283.368297] Lustre: server umount lustre-MDT0001 complete [ 2297.584658] Lustre: server umount lustre-OST0000 complete [ 2330.975953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.390127] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2331.439122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2331.838039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.690442] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2347.484808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.562452] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2347.619553] Lustre: 74530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2348.115919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2352.701774] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2360.713311] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2368.411352] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2377.220539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2377.509333] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2378.135288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.677529] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2393.499816] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2395.614515] Lustre: 77107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2397.990240] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2403.809050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2403.813942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.828454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2404.334934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2409.442225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2409.453416] Lustre: Skipped 3 previous similar messages [ 2409.744813] Lustre: server umount lustre-OST0000 complete [ 2414.565851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2414.566347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.601290] Lustre: Skipped 1 previous similar message [ 2419.680088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.688285] Lustre: Skipped 3 previous similar messages [ 2428.384285] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2429.923121] LustreError: 74534:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.944421] LustreError: 74534:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2430.615817] Lustre: server umount lustre-MDT0000 complete [ 2434.737849] LustreError: 77109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065444 with bad export cookie 10811608639885542376 [ 2434.743447] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2434.746186] LustreError: 77109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2435.009623] Lustre: server umount lustre-MDT0001 complete [ 2448.187326] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2452.136720] Key type lgssc unregistered [ 2452.610699] LNet: 78318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.622607] LNetError: 78318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2452.658215] LNet: Removed LNI 192.168.204.146@tcp [ 2454.098353] Key type .llcrypt unregistered [ 2454.101051] Key type ._llcrypt unregistered [ 2483.424672] Key type ._llcrypt registered [ 2483.427143] Key type .llcrypt registered [ 2483.513234] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2485.560088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2485.690444] alg: No test for adler32 (adler32-zlib) [ 2487.001681] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2487.430980] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2489.199793] Key type lgssc registered [ 2491.275132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2502.922858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2502.948304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.914416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2511.246495] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2514.401636] LustreError: 79731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.419309] LustreError: 79731:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2519.540161] LustreError: 79732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.612508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.605731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2528.175477] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2538.335170] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2545.992915] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2555.654798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2556.145365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.106783] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2572.351233] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2584.939913] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2586.094853] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2587.722827] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2590.874071] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2590.881387] Lustre: Skipped 1 previous similar message [ 2595.813544] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2595.820906] Lustre: Skipped 4 previous similar messages [ 2603.931727] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2603.938524] Lustre: Skipped 6 previous similar messages [ 2620.603835] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2620.607931] Lustre: Skipped 17 previous similar messages [ 2652.608899] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2652.615813] Lustre: Skipped 31 previous similar messages [ 2698.876216] Lustre: 87861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2702.472501] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:41:49 (1789065709) [ 2704.445278] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2706.920307] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:41:54 (1789065714) [ 2714.093134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.095318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2714.113695] Lustre: Skipped 1 previous similar message [ 2714.116510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2719.205168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2719.219845] Lustre: Skipped 2 previous similar messages [ 2724.323312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2724.331158] Lustre: Skipped 1 previous similar message [ 2726.367102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2726.543311] Lustre: server umount lustre-OST0000 complete [ 2733.025439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2733.045636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.072611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2734.574705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.680937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.685052] Lustre: Skipped 3 previous similar messages [ 2745.311376] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2745.729497] Lustre: server umount lustre-MDT0000 complete [ 2749.920637] LustreError: 79732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.935121] LustreError: 79732:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2750.458363] LustreError: 79712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065760 with bad export cookie 8485946811187312379 [ 2750.487799] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2750.489681] LustreError: 79712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2751.229523] Lustre: server umount lustre-MDT0001 complete [ 2765.280657] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2768.964534] Key type lgssc unregistered [ 2769.384789] LNet: 89112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2769.396337] LNetError: 89112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2769.446346] LNet: Removed LNI 192.168.204.146@tcp [ 2770.727114] Key type .llcrypt unregistered [ 2770.729115] Key type ._llcrypt unregistered [ 2776.666770] Key type ._llcrypt registered [ 2776.668385] Key type .llcrypt registered [ 2776.874738] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2782.819467] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules [ 2784.985607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2785.035172] alg: No test for adler32 (adler32-zlib) [ 2786.268957] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2786.537601] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2788.367284] Key type lgssc registered [ 2790.298657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2792.105693] LustreError: 90326:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2813.774430] LustreError: 90326:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2814.778532] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2826.513740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2826.561619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.368957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2833.714372] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2838.496511] LustreError: 91337:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2838.505820] LustreError: 91337:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2843.621891] LustreError: 91336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.998655] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:44:12 (1789065852) [ 2848.745838] LustreError: 91337:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.276550] LustreError: 91368:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2852.287594] LustreError: 91368:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2852.299179] LustreError: 91368:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 2852.785993] Lustre: server umount lustre-MDT0000 complete [ 2871.127441] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2875.022934] Key type lgssc unregistered [ 2875.404874] LNet: 92862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2875.416408] LNetError: 92862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2875.449390] LNet: Removed LNI 192.168.204.146@tcp [ 2876.469834] Key type .llcrypt unregistered [ 2876.478863] Key type ._llcrypt unregistered [ 2897.716377] Key type ._llcrypt registered [ 2897.718557] Key type .llcrypt registered [ 2897.837574] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2898.794432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2898.878205] alg: No test for adler32 (adler32-zlib) [ 2899.910435] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2900.234324] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2901.935200] Key type lgssc registered [ 2903.279181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2912.366031] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2912.426105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.161286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2919.356677] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2923.490864] LustreError: 94078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.503839] LustreError: 94078:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2928.613635] LustreError: 94079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.722727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.234894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2934.206410] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2942.439452] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2949.095734] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2959.722113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2960.549673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.589169] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2979.406984] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3043.972070] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:47:31 (1789066051) [ 3046.434641] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 3048.857326] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:47:36 (1789066056) [ 3056.099314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3056.106286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3056.138913] Lustre: Skipped 2 previous similar messages [ 3056.143548] Lustre: Skipped 1 previous similar message [ 3060.172264] Lustre: server umount lustre-MDT0000 complete [ 3061.226262] LustreError: 95067:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3061.253378] LustreError: 95067:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3065.117104] LustreError: 94059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066074 with bad export cookie 7181801931262893136 [ 3065.128993] LustreError: 94059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3065.130439] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3065.564950] Lustre: server umount lustre-MDT0001 complete [ 3079.606352] Lustre: server umount lustre-OST0000 complete [ 3093.684119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.172256] LustreError: 98397:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.421489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.624824] LustreError: 98398:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.950092] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3108.835293] LustreError: 98397:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3108.861461] LustreError: 98397:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3110.956731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.038447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3118.856421] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3128.965504] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3136.534575] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3152.570642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3152.769041] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3152.775696] LustreError: 98384:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3152.797602] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3152.807447] LustreError: 100484:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 3152.812966] LustreError: 100484:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 3152.818667] LustreError: 100484:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3152.910856] Lustre: server umount lustre-OST0000 complete [ 3152.913126] LustreError: 100484:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3163.540865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3163.841768] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3163.865174] LustreError: 98384:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3163.898702] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3163.906947] LustreError: 100950:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 3163.912619] LustreError: 100950:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 3163.918398] LustreError: 100950:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3164.021704] Lustre: server umount lustre-OST0000 complete [ 3164.025645] LustreError: 100950:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3175.393751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3175.540388] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3175.543256] Lustre: Found index 0 for lustre-OST0000, updating log [ 3175.547138] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3175.602436] Lustre: lustre-OST0000: new disk, initializing [ 3175.609310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3175.619441] Lustre: 101488:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3176.063978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3183.525539] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3188.107621] Lustre: Failing over lustre-OST0000 [ 3188.385823] Lustre: server umount lustre-OST0000 complete [ 3193.312298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3193.319771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.344900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3193.349830] Lustre: Skipped 1 previous similar message [ 3194.337623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.362503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3194.369858] Lustre: Skipped 1 previous similar message [ 3198.709636] Lustre: server umount lustre-MDT0000 complete [ 3199.456459] LustreError: 99039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.470152] LustreError: 99039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3202.886435] LustreError: 98377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066212 with bad export cookie 7181801931262895880 [ 3202.908150] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3202.912127] LustreError: 98377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3212.457512] Lustre: server umount lustre-MDT0001 complete [ 3223.369823] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:50:30 (1789066230) [ 3244.590951] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3257.605381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.049816] LustreError: 103836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.256130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.786479] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3273.214789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3279.086201] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3282.981716] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3291.184962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3291.974475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3291.983194] Lustre: Skipped 1 previous similar message [ 3291.993984] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3298.972533] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3299.186633] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3299.233967] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3299.241452] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3309.215510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3309.368199] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3309.448527] Lustre: lustre-OST0001: new disk, initializing [ 3309.452183] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3309.458399] Lustre: 106153:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3316.667263] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3317.910889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3317.923017] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3318.025703] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3325.699958] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3333.930840] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3340.567264] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 3340.771163] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3340.778292] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.626117] Lustre: server umount lustre-MDT0001 complete [ 3345.647049] LustreError: 103833:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.673774] LustreError: 103833:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3356.278352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.405903] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3356.412845] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3356.479494] Lustre: 103824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3357.211544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3357.217246] Lustre: Skipped 1 previous similar message [ 3358.222412] LustreError: 107725:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3358.234802] LustreError: 107725:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3358.244389] LustreError: 103828:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3358.348049] LustreError: 107736:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -17 [ 3358.360165] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3358.360165] [ 3361.976446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3362.286341] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3374.606221] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 14:53:01 (1789066381) [ 3396.106482] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3419.860253] Lustre: 109771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3450.459702] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3455.101687] Lustre: 110907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3551.713472] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.716387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3551.730990] Lustre: Skipped 4 previous similar messages [ 3551.756474] Lustre: Skipped 6 previous similar messages [ 3554.090484] Lustre: server umount lustre-MDT0000 complete [ 3556.320057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3556.328839] LustreError: 103837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.338102] LustreError: 103837:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 3558.050487] LustreError: 103817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066567 with bad export cookie 7181801931262896881 [ 3558.053886] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3558.058047] LustreError: 103817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3558.426563] Lustre: server umount lustre-MDT0001 complete [ 3571.518806] Lustre: server umount lustre-OST0000 complete [ 3585.936647] Lustre: server umount lustre-OST0001 complete [ 3592.799602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3604.350407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3615.065474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3627.129714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3649.507775] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3660.110723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.132524] Lustre: lustre-MDT0000: reset Object Index mappings [ 3660.695952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.670191] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3675.331184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3675.360918] Lustre: lustre-MDT0001: reset Object Index mappings [ 3681.583172] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3684.630329] Lustre: 116034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3691.805330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3698.714156] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3700.805351] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3706.881413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3708.328690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3714.566152] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3714.571028] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3716.431189] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3725.614575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3730.747591] Lustre: 117914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3788.734405] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:59:56 (1789066796) [ 3814.368475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3814.376987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3814.390197] Lustre: Skipped 1 previous similar message [ 3814.402205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3820.552633] Lustre: server umount lustre-MDT0000 complete [ 3822.052881] LustreError: 114880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3822.082249] LustreError: 114880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 3824.787185] LustreError: 114866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066834 with bad export cookie 7181801931262943788 [ 3824.803376] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3824.807396] LustreError: 114866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3831.394500] Lustre: server umount lustre-MDT0001 complete [ 3850.207436] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3850.389493] Lustre: server umount lustre-OST0000 complete [ 3860.750233] Lustre: server umount lustre-OST0001 complete [ 3867.807201] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3878.303734] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3922.210562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3922.452425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3922.525454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3922.630621] Lustre: lustre-MDT0000: new disk, initializing [ 3922.717519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3922.724100] Lustre: Skipped 3 previous similar messages [ 3922.737552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3927.973717] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3935.423934] Lustre: server umount lustre-MDT0000 complete [ 3945.372532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.625732] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3945.657246] Lustre: 122197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3945.683342] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3950.784140] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3956.421850] Lustre: server umount lustre-MDT0000 complete [ 3966.344499] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:02:53 (1789066973) [ 3968.088202] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3970.027772] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:02:57 (1789066977) [ 3986.572832] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3997.752592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4003.278231] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4012.497688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.629944] Lustre: 124249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4012.697492] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4012.699930] Lustre: Skipped 1 previous similar message [ 4012.791444] Lustre: lustre-MDT0001: new disk, initializing [ 4012.946675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4012.957613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4017.790898] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4023.206903] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4030.439138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4030.630962] Lustre: lustre-OST0000: new disk, initializing [ 4030.633935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4030.638675] Lustre: 125824:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4032.002225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4032.020664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4032.188092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4038.779822] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4050.100298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4050.306718] Lustre: lustre-OST0001: new disk, initializing [ 4050.312688] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4050.322061] Lustre: 126697:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4051.706623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4051.723066] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4051.817152] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4056.886517] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4067.514734] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4071.531073] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4505.945316] Lustre: 133123:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4505.955381] Lustre: 133123:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4505.961867] Lustre: 133123:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4505.967938] Lustre: 133123:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4505.975718] Lustre: 133123:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4505.982845] Lustre: 133123:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4848.758576] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:17:35 (1789067855) [ 4868.064639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4868.071978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4868.091774] Lustre: Skipped 5 previous similar messages [ 4868.100573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4868.108462] Lustre: Skipped 8 previous similar messages [ 4874.215680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4874.217857] Lustre: Skipped 4 previous similar messages [ 4879.327430] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4879.331504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4879.351034] Lustre: Skipped 3 previous similar messages [ 4879.623967] Lustre: server umount lustre-MDT0000 complete [ 4883.604983] LustreError: 133150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067893 with bad export cookie 7181801931263013445 [ 4883.613694] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4883.618627] LustreError: 133150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4884.449509] LustreError: 124255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.450122] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.459650] LustreError: 124255:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4884.469460] Lustre: Skipped 3 previous similar messages [ 4888.488288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4888.507698] Lustre: Skipped 6 previous similar messages [ 4890.200803] Lustre: server umount lustre-MDT0001 complete [ 4900.556450] Lustre: server umount lustre-OST0000 complete [ 4919.263145] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4919.691928] Lustre: server umount lustre-OST0001 complete [ 4928.230648] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4941.594506] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4983.989240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4984.300246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4984.349698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4984.472431] Lustre: lustre-MDT0000: new disk, initializing [ 4984.588548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4984.602521] Lustre: Skipped 5 previous similar messages [ 4984.616575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4990.546474] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5003.181962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.407550] Lustre: 136192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5003.489945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5003.495224] Lustre: Skipped 1 previous similar message [ 5003.602199] Lustre: lustre-MDT0001: new disk, initializing [ 5003.787789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5003.805734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5009.544220] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5019.521279] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5025.349448] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5032.630662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5033.029483] Lustre: lustre-OST0000: new disk, initializing [ 5033.039084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5033.051633] Lustre: 138160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5035.050493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5035.058297] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5035.189878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5040.882939] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5051.618252] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5063.073986] Lustre: lustre-MDD0000: changelog on [ 5066.002639] Lustre: lustre-MDD0001: changelog on [ 7249.489396] Lustre: lustre-MDD0001: changelog off [ 7251.553929] Lustre: 136198:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7252.583371] Lustre: lustre-MDD0000: changelog off [ 7262.696857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7262.705745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.709014] LustreError: Skipped 1 previous similar message [ 7262.710648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7262.728117] Lustre: Skipped 2 previous similar messages [ 7266.941300] Lustre: server umount lustre-OST0000 complete [ 7272.388723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.398833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7272.402853] Lustre: Skipped 1 previous similar message [ 7272.424255] Lustre: Skipped 2 previous similar messages [ 7277.547108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7277.564308] Lustre: Skipped 1 previous similar message [ 7278.819712] Lustre: server umount lustre-MDT0000 complete [ 7282.664090] LustreError: 136200:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.693337] LustreError: 136200:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7284.655677] LustreError: 136186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070294 with bad export cookie 7181801931263834853 [ 7284.662574] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7284.678674] LustreError: 136186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7285.210892] Lustre: server umount lustre-MDT0001 complete [ 7299.536906] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7302.884650] Key type lgssc unregistered [ 7303.170746] LNet: 147324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7303.181564] LNetError: 147324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7303.211381] LNet: Removed LNI 192.168.204.146@tcp [ 7304.239581] Key type .llcrypt unregistered [ 7304.243285] Key type ._llcrypt unregistered [ 7329.204948] Key type ._llcrypt registered [ 7329.208159] Key type .llcrypt registered [ 7329.414928] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7347.536967] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7349.063140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7349.136725] alg: No test for adler32 (adler32-zlib) [ 7350.257428] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 7350.660289] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7352.343909] Key type lgssc registered [ 7354.081831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7394.834235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7394.851679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7396.290317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7396.336312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7396.479295] Lustre: lustre-MDT0000: new disk, initializing [ 7396.622221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7396.666264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7402.362304] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7414.638872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.861785] Lustre: 150526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7414.949391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7414.956981] Lustre: Skipped 1 previous similar message [ 7415.057986] Lustre: lustre-MDT0001: new disk, initializing [ 7415.174305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7415.225577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7415.248219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7421.093115] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7432.050769] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7438.987954] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7447.348589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7447.736179] Lustre: lustre-OST0000: new disk, initializing [ 7447.746105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7447.758810] Lustre: 152494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7447.903541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7454.845588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7454.855482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7455.072417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7455.763589] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7467.601458] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7474.783185] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module obdecho/obdecho [ 7476.985059] LustreError: 153509:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7476.992772] LustreError: 153509:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7479.239288] LustreError: 153606:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7496.164149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7496.166123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7496.183890] Lustre: Skipped 1 previous similar message [ 7496.206450] Lustre: Skipped 2 previous similar messages [ 7499.005795] Lustre: server umount lustre-MDT0000 complete [ 7503.922927] LustreError: 150516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070513 with bad export cookie 16339602002575408053 [ 7503.931754] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7503.946554] LustreError: 150516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7504.327528] Lustre: server umount lustre-MDT0001 complete [ 7519.545648] Lustre: server umount lustre-OST0000 complete [ 7544.095888] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7555.158975] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7597.355298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7597.741823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7597.795216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7597.911227] Lustre: lustre-MDT0000: new disk, initializing [ 7597.997498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7598.019558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7602.975147] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7613.237246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7613.384123] Lustre: 157325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7613.478817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7613.481752] Lustre: Skipped 1 previous similar message [ 7613.543309] Lustre: lustre-MDT0001: new disk, initializing [ 7613.682747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7613.702955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7613.707729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7619.014406] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7631.380651] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7639.274792] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7646.620857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7647.041777] Lustre: lustre-OST0000: new disk, initializing [ 7647.050591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7647.054764] Lustre: 159298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7647.146174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7656.075636] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7657.079403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7657.117595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7657.247074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7668.330636] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7673.913731] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7674.127850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7679.964662] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7680.171453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7683.041663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7683.042539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.051647] LustreError: Skipped 1 previous similar message [ 7683.053114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7683.082598] Lustre: Skipped 2 previous similar messages [ 7688.167157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7688.172499] Lustre: Skipped 1 previous similar message [ 7688.847184] Lustre: server umount lustre-OST0000 complete [ 7693.290184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.299221] Lustre: Skipped 1 previous similar message [ 7693.301177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7693.311380] Lustre: Skipped 2 previous similar messages [ 7698.408106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7698.419764] Lustre: Skipped 1 previous similar message [ 7699.444635] Lustre: server umount lustre-MDT0000 complete [ 7703.521382] LustreError: 157337:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7703.551425] LustreError: 157337:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7703.616966] LustreError: 158123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070713 with bad export cookie 16339602002575409005 [ 7703.619123] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7703.629187] LustreError: 158123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7704.358138] Lustre: server umount lustre-MDT0001 complete [ 7708.581925] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 16:05:16 (1789070716) [ 7710.197933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7712.226569] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 16:05:19 (1789070719) [ 7721.726359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7722.221232] LustreError: 161709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7722.337467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7727.592476] LustreError: 161710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7727.696817] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7732.712445] LustreError: 161709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7737.830431] LustreError: 161710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.397911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7744.429859] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7753.578834] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7760.145787] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7768.527960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7769.117781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7769.125176] Lustre: Skipped 1 previous similar message [ 7777.067684] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7786.327357] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7805.408093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7805.414934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7805.429914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7809.326090] Lustre: server umount lustre-MDT0000 complete [ 7812.856940] LustreError: 164203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7813.402977] LustreError: 161689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070823 with bad export cookie 16339602002575409894 [ 7813.406487] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7813.418407] LustreError: 161689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7818.208661] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7818.226593] Lustre: Skipped 2 previous similar messages [ 7819.920482] Lustre: server umount lustre-MDT0001 complete [ 7827.781728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7844.646341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7844.825594] LustreError: 165578:0:(llog.c:1655:llog_backup()) MGC192.168.204.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7844.836286] Lustre: 165578:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7848.745220] LustreError: 165583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7848.775100] LustreError: 165583:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7884.779928] LustreError: 165584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7884.794791] LustreError: 165584:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 28 previous similar messages [ 7905.759615] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7905.767938] Lustre: 165578:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7946.761889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7950.073249] LustreError: 165585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7950.100187] LustreError: 165585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 45 previous similar messages [ 7951.863633] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7951.964309] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7962.573876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7971.303784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7985.632143] LustreError: 148745:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a1dcc4ef x1875976241952128/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7991.090274] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7991.283826] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8001.276346] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8003.666965] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8012.181598] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8014.243926] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8037.354222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8037.394407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8037.410527] Lustre: Skipped 7 previous similar messages [ 8043.004569] Lustre: server umount lustre-MDT0000 complete [ 8047.797401] LustreError: 161691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071057 with bad export cookie 16339602002575416383 [ 8047.798430] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8047.821570] LustreError: 161691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8048.221485] Lustre: server umount lustre-MDT0001 complete [ 8063.434119] Lustre: server umount lustre-OST0000 complete [ 8089.345960] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8102.818956] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8151.249536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8151.573789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8151.602388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8151.693979] Lustre: lustre-MDT0000: new disk, initializing [ 8151.773133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8151.781322] Lustre: Skipped 1 previous similar message [ 8151.806084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8156.850163] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8169.095120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8169.295872] Lustre: 170688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8169.674853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8175.069594] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8185.801937] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8192.000786] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8199.955520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8200.291680] Lustre: lustre-OST0000: new disk, initializing [ 8200.301386] Lustre: Skipped 1 previous similar message [ 8200.310712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8200.317558] Lustre: Skipped 2 previous similar messages [ 8200.333710] Lustre: 172658:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8205.922835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8205.933353] Lustre: Skipped 1 previous similar message [ 8205.949097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8206.109212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8207.459159] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8219.080787] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8225.271372] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8225.535725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8229.712206] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8229.928926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8237.029629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.035120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8237.041605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8237.041615] Lustre: Skipped 4 previous similar messages [ 8237.055613] Lustre: Skipped 3 previous similar messages [ 8246.246408] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8246.515742] Lustre: server umount lustre-OST0000 complete [ 8251.873836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8257.435213] Lustre: server umount lustre-MDT0000 complete [ 8257.505201] LustreError: 170702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8257.521879] LustreError: 170702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 34 previous similar messages [ 8262.181147] LustreError: 170682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071271 with bad export cookie 16339602002575433190 [ 8262.184231] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8262.194272] LustreError: 170682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8262.705639] Lustre: server umount lustre-MDT0001 complete [ 8275.011702] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 16:14:42 (1789071282) [ 8283.455752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.399868] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8300.489487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8306.379675] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8315.936927] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8323.403034] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8331.168985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8338.606271] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8349.761172] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8369.370888] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 16:16:16 (1789071376) [ 8374.762972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8374.774109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.790449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8374.814200] Lustre: Skipped 3 previous similar messages [ 8374.830556] Lustre: Skipped 7 previous similar messages [ 8380.174802] Lustre: server umount lustre-OST0000 complete [ 8399.332966] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8400.049276] Lustre: server umount lustre-MDT0000 complete [ 8404.678620] LustreError: 175050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071414 with bad export cookie 16339602002575434072 [ 8404.684277] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8404.690324] LustreError: 175050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8405.389806] Lustre: server umount lustre-MDT0001 complete [ 8418.289294] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8421.625849] Key type lgssc unregistered [ 8422.006601] LNet: 178825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8422.017824] LNetError: 178825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8422.041874] LNet: Removed LNI 192.168.204.146@tcp [ 8422.920438] Key type .llcrypt unregistered [ 8422.922234] Key type ._llcrypt unregistered [ 8447.145319] Key type ._llcrypt registered [ 8447.147988] Key type .llcrypt registered [ 8447.269615] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8448.608351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8448.806199] alg: No test for adler32 (adler32-zlib) [ 8449.905478] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 8450.278226] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8452.022149] Key type lgssc registered [ 8453.196672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8462.584370] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8462.593557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8478.113418] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8483.231698] LustreError: 180069:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 8488.351468] LustreError: 180069:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8488.359924] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8488.365610] LustreError: MGC192.168.204.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8488.379945] LustreError: 180069:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8488.394677] LustreError: 180069:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 8488.408153] LustreError: 180069:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8488.454133] Lustre: server umount lustre-OST0000 complete [ 8488.456508] LustreError: 180069:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8498.928461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8500.623941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8506.438629] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8510.949846] LustreError: 180568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8510.969704] LustreError: 180568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8515.749378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8516.064232] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8516.358036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8521.661340] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8531.047163] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8537.660387] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8544.792641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8544.995234] LustreError: 182385:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8545.006685] Lustre: 182385:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8545.199766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8552.460916] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8561.922211] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8565.731492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8565.739279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8565.763842] Lustre: Skipped 2 previous similar messages [ 8565.770292] Lustre: Skipped 1 previous similar message [ 8570.572395] Lustre: server umount lustre-MDT0000 complete [ 8570.856715] LustreError: 180564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8570.877970] LustreError: 180564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8574.605329] LustreError: 180548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071584 with bad export cookie 13498037088819756747 [ 8574.609117] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8574.625467] LustreError: 180548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8574.950610] Lustre: server umount lustre-MDT0001 complete [ 8589.316978] Lustre: server umount lustre-OST0000 complete [ 8602.382485] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8617.964186] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8656.089434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8656.323665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8656.345401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8656.398765] Lustre: lustre-MDT0000: new disk, initializing [ 8656.446044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8656.467743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8660.957130] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8670.807526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.898152] Lustre: 185929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8670.954889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8670.958068] Lustre: Skipped 1 previous similar message [ 8671.001429] Lustre: lustre-MDT0001: new disk, initializing [ 8671.059230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8671.080839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8671.090224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8675.114830] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8686.933584] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8693.736974] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8701.605189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8702.239881] Lustre: lustre-OST0000: new disk, initializing [ 8702.251530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8702.259798] Lustre: 187899:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8702.410896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8709.209410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8709.229776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8709.450098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8710.153278] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8723.482503] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8729.046893] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8729.211554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8733.420212] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8733.680544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8740.328740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8740.341670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8740.343095] LustreError: Skipped 1 previous similar message [ 8740.344562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8740.359214] Lustre: Skipped 1 previous similar message [ 8740.362756] Lustre: Skipped 1 previous similar message [ 8742.931770] Lustre: server umount lustre-OST0000 complete [ 8748.001660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8748.014802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8748.048588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8750.566454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8753.476290] Lustre: server umount lustre-MDT0000 complete [ 8755.680816] LustreError: 185943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8755.702589] LustreError: 185943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8757.285232] LustreError: 185924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071767 with bad export cookie 13498037088819757468 [ 8757.294610] LustreError: 185924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8757.299150] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8757.701490] Lustre: server umount lustre-MDT0001 complete [ 8768.653596] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 16:22:55 (1789071775) [ 8793.018693] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8799.144854] Key type lgssc unregistered [ 8799.768773] LNet: 190845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8799.790371] LNetError: 190845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8799.845082] LNet: Removed LNI 192.168.204.146@tcp [ 8801.386261] Key type .llcrypt unregistered [ 8801.397070] Key type ._llcrypt unregistered [ 8823.144995] Key type ._llcrypt registered [ 8823.146550] Key type .llcrypt registered [ 8823.236350] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8824.275653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8824.308207] alg: No test for adler32 (adler32-zlib) [ 8825.421235] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 8825.701281] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8827.400203] Key type lgssc registered [ 8828.823285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8838.331740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8838.346667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8839.848574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8846.457673] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8849.384423] LustreError: 192061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8849.421995] LustreError: 192061:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8854.503043] LustreError: 192062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8859.642953] LustreError: 192061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8861.585163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8862.228867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8868.081608] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8877.188264] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8884.319445] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8892.329612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8892.708650] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8892.713592] LustreError: 193926:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8892.721284] Lustre: 193926:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8892.817836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8900.915761] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8909.858801] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8913.385300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8913.396685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.408277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8914.401331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8914.404079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8914.423943] Lustre: Skipped 1 previous similar message [ 8918.367843] Lustre: server umount lustre-MDT0000 complete [ 8919.522451] LustreError: 192061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8919.558727] LustreError: 192061:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8921.899652] LustreError: 192043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071931 with bad export cookie 15770340966774075621 [ 8921.905540] LustreError: 192043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8921.905772] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8922.221862] Lustre: server umount lustre-MDT0001 complete [ 8936.199604] Lustre: server umount lustre-OST0000 complete [ 8949.551634] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8961.954808] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9008.168956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9008.372928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9008.413751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9008.469812] Lustre: lustre-MDT0000: new disk, initializing [ 9008.527039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9008.534489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9015.253454] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9030.079186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9030.209905] Lustre: 197455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9030.279118] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9030.285877] Lustre: Skipped 1 previous similar message [ 9030.289399] LustreError: 198238:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 9030.295356] LustreError: 198238:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 9030.304554] Lustre: 198238:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 9030.320643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9030.324349] Lustre: Skipped 1 previous similar message [ 9030.412508] Lustre: lustre-MDT0001: new disk, initializing [ 9030.605065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9030.700190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9030.707335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9036.869037] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9047.824597] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9055.152329] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9064.361957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9064.634880] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9064.639326] Lustre: Skipped 1 previous similar message [ 9064.644624] LustreError: 199400:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 9064.660366] LustreError: 199400:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 9064.678505] Lustre: 199400:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 9064.704444] Lustre: 199400:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 9064.743159] Lustre: lustre-OST0000: new disk, initializing [ 9064.752495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9064.770033] Lustre: 199424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9064.966057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9071.206615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9071.218764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9071.471232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9072.917741] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9086.268280] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9092.470599] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9092.827613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9101.028922] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9101.340338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9107.425932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9107.429400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9107.437209] Lustre: Skipped 1 previous similar message [ 9107.440475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9107.449705] Lustre: Skipped 3 previous similar messages [ 9112.554658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9112.573152] Lustre: Skipped 1 previous similar message [ 9117.675459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9117.688847] Lustre: Skipped 1 previous similar message [ 9118.176969] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9118.432774] Lustre: server umount lustre-OST0000 complete [ 9122.790152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9122.800638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9122.801873] LustreError: Skipped 1 previous similar message [ 9127.905537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9127.916170] Lustre: Skipped 3 previous similar messages [ 9137.123594] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9137.350901] Lustre: server umount lustre-MDT0000 complete [ 9138.148608] LustreError: 200766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9138.197728] LustreError: 200766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9141.681904] LustreError: 197446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072151 with bad export cookie 15770340966774076349 [ 9141.694951] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9141.698962] LustreError: 197446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9141.974266] Lustre: server umount lustre-MDT0001 complete [ 9159.257726] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:29:25 (1789072165) [ 9181.924515] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9193.643676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9194.248492] LustreError: 202560:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9194.302828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9198.317618] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9208.187739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9214.144390] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9218.757215] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9226.444878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9226.834626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9226.842106] Lustre: Skipped 1 previous similar message [ 9233.552400] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9243.349298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9243.517393] Lustre: lustre-OST0001: new disk, initializing [ 9243.520628] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9243.525985] Lustre: 204868:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9244.070496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9244.083735] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9244.156524] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9249.509313] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9266.759792] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9276.267448] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9278.955903] Lustre: DEBUG MARKER: ADD OST3 [ 9285.244599] loop: module loaded [ 9285.347536] loop0: detected capacity change from 0 to 204800000 [ 9285.627330] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9285.644671] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9285.657697] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9292.641323] loop0: detected capacity change from 0 to 204800000 [ 9293.107648] Lustre: DEBUG MARKER: STOP OST3 [ 9300.558504] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9300.982884] Lustre: lustre-OST0003: new disk, initializing [ 9300.990471] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 9301.005638] Lustre: 206913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9301.372213] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 9301.385165] Lustre: Skipped 1 previous similar message [ 9302.498074] Lustre: *** cfs_fail_loc=2109, val=2*** [ 9302.752108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 9302.777059] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 9302.850595] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 9304.544050] LustreError: 207003:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 9306.142320] Lustre: Failing over lustre-OST0003 [ 9306.446129] Lustre: server umount lustre-OST0003 complete [ 9307.953463] LustreError: 204909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9308.007233] LustreError: 204909:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9308.140885] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9308.146310] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9308.164066] Lustre: Skipped 1 previous similar message [ 9312.436528] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9321.752530] Lustre: DEBUG MARKER: START OST3 again [ 9327.351535] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9327.595382] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 9328.361645] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9328.854052] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9328.856157] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9333.514463] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9358.133033] Lustre: Failing over lustre-OST0003 [ 9358.222664] Lustre: server umount lustre-OST0003 complete [ 9358.332481] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9358.358050] Lustre: Skipped 1 previous similar message [ 9358.378750] LustreError: 204050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9358.396553] LustreError: 204050:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 9378.791416] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9378.798596] Lustre: Skipped 4 previous similar messages [ 9378.800211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9378.807317] Lustre: Skipped 5 previous similar messages [ 9384.287639] Lustre: server umount lustre-MDT0000 complete [ 9389.955754] LustreError: 202542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072399 with bad export cookie 15770340966774077231 [ 9389.956898] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9389.963051] LustreError: 202542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9404.383204] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9404.810475] Lustre: server umount lustre-MDT0001 complete [ 9409.951133] Lustre: server umount lustre-OST0000 complete [ 9414.821825] Lustre: server umount lustre-OST0001 complete [ 9423.422207] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9433.453637] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9476.795182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9476.974581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9477.017788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9477.072243] Lustre: lustre-MDT0000: new disk, initializing [ 9477.143908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9477.148379] Lustre: Skipped 1 previous similar message [ 9477.168379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9480.908552] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9493.108174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9493.366651] Lustre: 212337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9493.509781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9493.514753] Lustre: Skipped 1 previous similar message [ 9493.674482] Lustre: lustre-MDT0001: new disk, initializing [ 9493.933139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9493.955067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9499.296584] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9511.133881] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9518.502741] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9525.949766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9526.188157] Lustre: lustre-OST0000: new disk, initializing [ 9526.191690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9526.196334] Lustre: 214304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9527.646975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9527.652336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9527.727605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9532.338097] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9543.248916] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9549.051738] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9549.255863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9555.846214] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9556.028644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9563.619079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9563.626670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9563.635355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9563.635407] Lustre: Skipped 3 previous similar messages [ 9563.654606] Lustre: Skipped 11 previous similar messages [ 9573.355018] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9573.525182] Lustre: server umount lustre-OST0000 complete [ 9584.632488] Lustre: server umount lustre-MDT0000 complete [ 9588.730670] LustreError: 212329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072598 with bad export cookie 15770340966774079527 [ 9588.734273] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9588.758687] LustreError: 212329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9589.020771] Lustre: server umount lustre-MDT0001 complete [ 9592.366911] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:36:39 (1789072599) [ 9611.035992] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9626.601964] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9677.129776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9677.440550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9677.495440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9677.579232] Lustre: lustre-MDT0000: new disk, initializing [ 9677.702240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9683.177694] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9699.506720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9699.721022] Lustre: 218943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9700.020582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9705.989337] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9717.653151] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9725.144148] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9734.392885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9734.936262] Lustre: 220915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9735.075072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9735.078363] Lustre: Skipped 4 previous similar messages [ 9736.816931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9737.053335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9742.989949] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9753.872935] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9760.019416] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9760.266133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9770.044786] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9770.467109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9778.146480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9778.153948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9778.162871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9778.162880] LustreError: Skipped 1 previous similar message [ 9778.171744] Lustre: Skipped 3 previous similar messages [ 9778.212175] Lustre: Skipped 7 previous similar messages [ 9779.666868] Lustre: server umount lustre-OST0000 complete [ 9791.296457] Lustre: server umount lustre-MDT0000 complete [ 9793.510326] LustreError: 218956:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9793.528335] LustreError: 218956:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [ 9796.071517] LustreError: 220914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072805 with bad export cookie 15770340966774080409 [ 9796.077166] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9796.082647] LustreError: 220914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9796.628808] Lustre: server umount lustre-MDT0001 complete [ 9807.813301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9816.052635] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9830.165549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9836.806715] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9847.376360] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9855.961255] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9866.191395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9876.381627] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9888.599606] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9902.049972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9902.062646] LustreError: Skipped 2 previous similar messages [ 9906.914338] Lustre: server umount lustre-OST0000 complete [ 9912.800205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9912.807576] Lustre: Skipped 4 previous similar messages [ 9912.816273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9912.821817] Lustre: Skipped 7 previous similar messages [ 9917.328133] Lustre: server umount lustre-MDT0000 complete [ 9922.386581] LustreError: 223117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072932 with bad export cookie 15770340966774081291 [ 9922.396657] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9922.401036] LustreError: 223117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9922.840839] Lustre: server umount lustre-MDT0001 complete [10032.133499] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10035.374509] Key type lgssc unregistered [10035.903736] LNet: 226805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10035.920729] LNetError: 226805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10035.968681] LNet: Removed LNI 192.168.204.146@tcp [10037.157605] Key type .llcrypt unregistered [10037.159116] Key type ._llcrypt unregistered [10072.748492] Key type ._llcrypt registered [10072.758546] Key type .llcrypt registered [10072.859277] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10094.011858] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10096.335730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10096.370776] alg: No test for adler32 (adler32-zlib) [10097.709752] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [10098.195227] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10099.929840] Key type lgssc registered [10102.581825] Lustre: Echo OBD driver; http://www.lustre.org/ [10156.543585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10156.585812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10158.084891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10158.125372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10158.241879] Lustre: lustre-MDT0000: new disk, initializing [10158.349943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10158.388771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10164.259237] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10180.507533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.695462] Lustre: 230157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10180.819206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10180.832478] Lustre: Skipped 1 previous similar message [10180.919324] Lustre: lustre-MDT0001: new disk, initializing [10181.057793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10181.106189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10181.117871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10187.041490] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10198.003693] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10205.424226] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10213.426449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10213.645976] Lustre: lustre-OST0000: new disk, initializing [10213.649073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10213.657450] Lustre: 232128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10213.757289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10221.916335] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10222.782203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10222.806206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10222.990554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10234.978729] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10242.104960] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10242.512451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10249.981444] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10250.467114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10253.802066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10253.806051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10253.861795] Lustre: Skipped 1 previous similar message [10253.869789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10253.880020] Lustre: Skipped 1 previous similar message [10258.918842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10258.940649] Lustre: Skipped 1 previous similar message [10264.044747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10264.062372] Lustre: Skipped 1 previous similar message [10267.103893] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10267.395633] Lustre: server umount lustre-OST0000 complete [10273.249189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10273.265363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10273.289774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10274.283926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10279.868615] Lustre: server umount lustre-MDT0000 complete [10283.798796] LustreError: 230148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073293 with bad export cookie 11965719447887867386 [10283.810275] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10283.812570] LustreError: 230148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10284.161540] Lustre: server umount lustre-MDT0001 complete [10313.839198] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10324.690756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.135116] LustreError: 235090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.172557] LustreError: 235090:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10325.252545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10330.595641] LustreError: 235091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10331.849585] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10334.693169] LustreError: 235090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10339.843327] LustreError: 235090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10344.928070] LustreError: 235091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10345.237618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10346.129209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10355.612784] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10360.542549] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10369.261543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10369.921730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10378.797892] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10390.139983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10390.508586] Lustre: lustre-OST0001: new disk, initializing [10390.515727] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10390.530453] Lustre: 237400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10390.753805] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10396.373202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10396.381200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10396.528647] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10398.690919] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10412.258446] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10423.567650] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10438.863180] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:50:45 (1789073445) [10442.721954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10442.738860] Lustre: Skipped 3 previous similar messages [10442.741356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10442.756175] Lustre: Skipped 5 previous similar messages [10447.844030] LustreError: 235091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10447.878925] LustreError: 235091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10448.087838] Lustre: server umount lustre-MDT0000 complete [10452.957646] LustreError: 238172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073462 with bad export cookie 11965719447887868268 [10452.983802] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10452.984859] LustreError: 238172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10453.038307] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10453.060298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10453.067281] Lustre: Skipped 1 previous similar message [10453.779744] Lustre: server umount lustre-MDT0001 complete [10459.493817] Lustre: server umount lustre-OST0000 complete [10464.940728] Lustre: server umount lustre-OST0001 complete [10473.876560] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10487.334416] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10531.304855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10531.599414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10531.664879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10531.858802] Lustre: lustre-MDT0000: new disk, initializing [10531.994138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10532.010699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10538.193546] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10550.563978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10550.680736] Lustre: 241349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10550.745990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10550.749310] Lustre: Skipped 1 previous similar message [10550.851710] Lustre: lustre-MDT0001: new disk, initializing [10551.089035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10551.107802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10556.846240] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10567.730234] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10575.462961] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10583.417107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10583.703661] Lustre: lustre-OST0000: new disk, initializing [10583.708513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10583.716945] Lustre: 243319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10585.339648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10585.350644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10585.562224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10590.194129] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10602.413502] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10609.586772] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10609.937702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10616.418361] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10616.637272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10621.433839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10621.434158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10621.435081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10621.444889] Lustre: Skipped 2 previous similar messages [10621.491989] LustreError: Skipped 1 previous similar message [10633.185763] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10633.426125] Lustre: server umount lustre-OST0000 complete [10638.311708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10638.322555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10644.557917] Lustre: server umount lustre-MDT0000 complete [10647.008973] LustreError: 241354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10647.025891] LustreError: 241354:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10648.817295] LustreError: 241340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073658 with bad export cookie 11965719447887870403 [10648.834641] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10649.195778] Lustre: server umount lustre-MDT0001 complete [10660.110558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10660.901970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10660.907601] Lustre: Skipped 2 previous similar messages [10666.100332] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10676.370060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10683.325634] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10693.647840] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10702.698467] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10716.356178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10726.784116] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10736.251485] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10885.600836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10885.608076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10885.617350] Lustre: Skipped 1 previous similar message [10885.622402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10885.626985] Lustre: Skipped 8 previous similar messages [10891.326105] Lustre: server umount lustre-OST0000 complete [10897.377834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10897.397600] LustreError: Skipped 1 previous similar message [10901.691344] Lustre: server umount lustre-MDT0000 complete [10905.059406] LustreError: 245539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10905.078876] LustreError: 245539:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10905.602329] LustreError: 246273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073915 with bad export cookie 11965719447887871292 [10905.605325] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10905.615040] LustreError: 246273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10906.259288] Lustre: server umount lustre-MDT0001 complete [11006.828890] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11010.353747] Key type lgssc unregistered [11010.711688] LNet: 249147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11010.719173] LNetError: 249147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11010.736895] LNet: Removed LNI 192.168.204.146@tcp [11011.675736] Key type .llcrypt unregistered [11011.681074] Key type ._llcrypt unregistered [11042.888466] Key type ._llcrypt registered [11042.891195] Key type .llcrypt registered [11042.993136] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11064.825715] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11066.601836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11066.741619] alg: No test for adler32 (adler32-zlib) [11068.097199] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [11068.349742] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11069.999678] Key type lgssc registered [11071.084433] Lustre: Echo OBD driver; http://www.lustre.org/ [11127.549461] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11139.609202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11139.649881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11141.002579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11141.054301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11141.185604] Lustre: lustre-MDT0000: new disk, initializing [11141.381229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11141.477297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11145.863993] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11160.504843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11160.638164] Lustre: 253230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11160.733168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11160.739676] Lustre: Skipped 1 previous similar message [11160.928066] Lustre: lustre-MDT0001: new disk, initializing [11161.132151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11161.188572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11161.214334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11167.228222] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11174.181602] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11183.577866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11184.036364] Lustre: lustre-OST0000: new disk, initializing [11184.044093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11184.060166] Lustre: 254865:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11184.263337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11192.750322] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11193.511127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11193.529512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11193.772264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11205.697740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11206.175047] Lustre: lustre-OST0001: new disk, initializing [11206.183790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11206.196903] Lustre: 255736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11206.323735] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11213.803169] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11214.491526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11214.503266] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11214.642374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11226.930627] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11236.535522] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11250.656977] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11250.660072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11250.686439] Lustre: Skipped 3 previous similar messages [11250.701044] Lustre: Skipped 3 previous similar messages [11255.777316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11255.782646] Lustre: Skipped 2 previous similar messages [11256.314474] Lustre: server umount lustre-MDT0000 complete [11260.667284] LustreError: 254026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074270 with bad export cookie 10647442096720506317 [11260.672401] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11260.677385] LustreError: 254026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11260.901106] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.046802] Lustre: server umount lustre-MDT0001 complete [11268.147401] Lustre: server umount lustre-OST0000 complete [11271.662273] Lustre: server umount lustre-OST0001 complete [11290.064967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11301.033307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11310.205202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11322.431124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11344.737924] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11355.498027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11355.524357] Lustre: lustre-MDT0000: reset Object Index mappings [11355.934280] LustreError: 260752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11355.948256] LustreError: 260752:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11356.013475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11359.868621] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11361.257210] LustreError: 260753:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11365.356610] LustreError: 260752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11368.809345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11368.843890] Lustre: lustre-MDT0001: reset Object Index mappings [11369.484048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11374.691184] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11377.959777] Lustre: 261895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11384.635058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11391.098883] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11395.253391] LustreError: 262250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11400.438224] LustreError: 262251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11400.453201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [11400.460286] LustreError: 262251:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11400.692604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11401.070130] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11401.078557] Lustre: Skipped 1 previous similar message [11406.338133] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [11408.492936] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11417.729465] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11422.097991] Lustre: 263766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11430.881150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11430.888338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11430.906506] Lustre: Skipped 1 previous similar message [11430.916261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11434.361935] Lustre: server umount lustre-MDT0000 complete [11437.028279] LustreError: 262021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11437.047282] LustreError: 262021:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11437.990878] LustreError: 263768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074447 with bad export cookie 10647442096720513107 [11437.998942] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11438.006996] LustreError: 263768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11438.367910] Lustre: server umount lustre-MDT0001 complete [11452.606500] Lustre: server umount lustre-OST0000 complete [11465.536330] Lustre: server umount lustre-OST0001 complete [11484.786960] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 17:08:12 (1789074492) [11503.273884] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11514.141290] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11556.511853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11556.937736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11556.961405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11557.038473] Lustre: lustre-MDT0000: new disk, initializing [11557.163432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11557.193881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11563.192552] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11575.522551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11575.702852] Lustre: 268022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11575.773894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11575.777671] Lustre: Skipped 1 previous similar message [11575.879807] Lustre: lustre-MDT0001: new disk, initializing [11576.049825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11576.062958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11581.210965] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11592.740191] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11599.082659] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11607.366493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11607.847407] Lustre: lustre-OST0000: new disk, initializing [11607.854329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11607.870939] Lustre: 269993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11609.037614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11609.055312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11609.319347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11615.161562] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11625.362938] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11629.408131] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11629.592818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11634.981522] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11635.145139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11639.783266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11639.785370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11639.825849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11639.836195] Lustre: Skipped 4 previous similar messages [11639.840144] Lustre: Skipped 4 previous similar messages [11643.385579] Lustre: server umount lustre-OST0000 complete [11650.030769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11650.042209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11650.053758] Lustre: Skipped 1 previous similar message [11650.058659] Lustre: Skipped 1 previous similar message [11654.701908] Lustre: server umount lustre-MDT0000 complete [11655.139900] LustreError: 268030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11655.166078] LustreError: 268030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11658.918307] LustreError: 268017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074668 with bad export cookie 10647442096720515193 [11658.927723] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11658.931419] LustreError: 268017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11659.308160] Lustre: server umount lustre-MDT0001 complete [11680.683451] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11691.819411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.093983] LustreError: 272930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11692.144429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11692.153450] Lustre: Skipped 2 previous similar messages [11696.399435] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11706.364491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11712.190208] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11715.560804] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11722.990733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11730.064092] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11739.757342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11739.975435] Lustre: lustre-OST0001: new disk, initializing [11739.979347] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11739.985078] Lustre: 275238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11741.866650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11741.879315] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11741.963533] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11747.256935] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11760.169604] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11766.127900] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11771.880516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11772.203753] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11783.066617] Lustre: server umount lustre-MDT0000 complete [11783.147018] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11783.178415] LustreError: 275373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11783.193432] LustreError: 275373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11787.738736] LustreError: 273665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074797 with bad export cookie 10647442096720516082 [11787.740654] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11787.758590] LustreError: 273665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11788.266327] Lustre: server umount lustre-MDT0001 complete [11802.542500] Lustre: server umount lustre-OST0000 complete [11804.455351] Lustre: 250734:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789074798/real 1789074798] req@ffff962dea3a7100 x1875980140649984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789074814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11804.518732] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11804.555341] Lustre: Skipped 3 previous similar messages [11807.320760] Lustre: server umount lustre-OST0001 complete [11828.818953] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11842.265492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11842.903210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11842.913021] Lustre: Skipped 3 previous similar messages [11848.907723] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11859.651342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11865.263407] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11869.261923] Lustre: 279720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11878.873061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11887.415131] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11887.449877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11896.366577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11901.957082] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11902.831645] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11910.660598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11914.730212] Lustre: 281599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11920.642335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11920.818671] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11934.194666] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 17:15:41 (1789074941) [11952.106785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11952.117048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11952.142927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11952.147755] Lustre: Skipped 1 previous similar message [11956.488244] Lustre: server umount lustre-MDT0000 complete [11958.258240] LustreError: 278581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11958.294220] LustreError: 278581:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [11960.906379] LustreError: 278563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074970 with bad export cookie 10647442096720522501 [11960.925321] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11960.925558] LustreError: 278563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11967.528635] Lustre: server umount lustre-MDT0001 complete [11986.399118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11986.629933] Lustre: server umount lustre-OST0000 complete [11991.584958] Lustre: server umount lustre-OST0001 complete [11998.968753] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12009.574697] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12063.178251] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12074.659469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12074.931613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12074.969521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12075.082730] Lustre: lustre-MDT0000: new disk, initializing [12075.164898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12080.212938] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12094.605625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12094.734428] Lustre: 286131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12094.965524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12101.996059] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12109.751733] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12118.510470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12118.712813] Lustre: 287766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12118.769927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12118.776222] Lustre: Skipped 5 previous similar messages [12120.525583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12126.615642] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12138.512291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12138.762883] Lustre: 288637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12139.791183] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12139.811495] Lustre: Skipped 1 previous similar message [12139.955621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12145.308088] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12156.505979] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12181.576574] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 17:19:48 (1789075188) [12184.544302] Lustre: server umount lustre-MDT0000 complete [12186.081425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12186.090566] Lustre: Skipped 6 previous similar messages [12188.673742] LustreError: 287765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075198 with bad export cookie 10647442096720530418 [12188.679912] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12188.684355] LustreError: 287765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12188.934440] Lustre: server umount lustre-MDT0001 complete [12203.436674] Lustre: server umount lustre-OST0000 complete [12206.553971] Lustre: 250734:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789075200/real 1789075200] req@ffff962dc5ab2d80 x1875980140808832/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789075216 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12207.794583] Lustre: server umount lustre-OST0001 complete [12218.189576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12218.424061] LustreError: 290881:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12218.435254] LustreError: 290881:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [12224.492663] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12235.392058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12239.773666] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12249.814246] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12257.125851] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12265.775360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12273.737609] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12282.511598] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12313.722201] loop1: detected capacity change from 0 to 204800000 [12314.045619] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12314.082151] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12314.105836] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12320.739436] loop1: detected capacity change from 0 to 204800000 [12328.237907] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12328.341176] LustreError: 290869:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs16226: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [12328.370560] LustreError: 290869:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs16226 [12328.379928] LustreError: 290869:0:(mgs_handler.c:517:mgs_target_reg()) fs16226-MDT0001: Error checking target index: rc = -13 [12328.397311] LustreError: fs16226-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12328.411928] LustreError: 294623:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [12328.417852] LustreError: 294623:0:(tgt_mount.c:2128:server_put_super()) no obd fs16226-MDT0001 [12328.421683] LustreError: 294623:0:(tgt_mount.c:116:server_deregister_mount()) fs16226-MDT0001 not registered [12328.460263] Lustre: server umount fs16226-MDT0001 complete [12328.463069] LustreError: 294623:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12340.667482] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12340.830398] Lustre: Setting parameter fs16226-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs16226-MDT0001 [12340.848268] Lustre: Skipped 1 previous similar message [12340.924270] Lustre: srv-fs16226-MDT0001: No data found on store. Initialize space. [12340.930828] Lustre: Skipped 4 previous similar messages [12341.020831] Lustre: fs16226-MDT0001: new disk, initializing [12341.027257] Lustre: Skipped 3 previous similar messages [12341.178813] Lustre: srv-fs16226-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12341.208025] Lustre: 295091:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16226-MDT0001-mdtlov: can't create new llog: rc = -115 [12341.219868] LustreError: 295091:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16226-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12347.185843] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12354.997526] Lustre: server umount fs16226-MDT0001 complete [12360.689682] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12365.820596] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12371.826736] loop1: detected capacity change from 0 to 204800000 [12379.229189] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12379.870856] Lustre: 296659:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16226-MDT0001-mdtlov: can't create new llog: rc = -115 [12379.875857] LustreError: 296659:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16226-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12387.058975] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12394.808922] Lustre: server umount fs16226-MDT0001 complete [12409.825763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12409.833226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12409.842673] Lustre: Skipped 2 previous similar messages [12409.849502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12409.866143] Lustre: Skipped 8 previous similar messages [12414.485554] Lustre: server umount lustre-MDT0000 complete [12424.907139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12425.202224] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12430.592272] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12430.882259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12440.260757] loop1: detected capacity change from 0 to 204800000 [12447.047226] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12447.709645] Lustre: 299108:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16226-MDT0001-mdtlov: can't create new llog: rc = -115 [12447.744497] LustreError: 299108:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16226-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12454.699183] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12463.650738] Lustre: server umount fs16226-MDT0001 complete [12497.393099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12497.412468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12497.426098] Lustre: Skipped 5 previous similar messages [12502.410489] Lustre: server umount lustre-OST0000 complete [12513.946355] Lustre: server umount lustre-MDT0000 complete [12518.684300] LustreError: 290863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075528 with bad export cookie 10647442096720533099 [12518.689666] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12518.714251] LustreError: 290863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12519.235543] Lustre: server umount lustre-MDT0001 complete [12533.789657] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12537.990642] Key type lgssc unregistered [12538.368640] LNet: 301455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12538.388232] LNetError: 301455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12538.430551] LNet: Removed LNI 192.168.204.146@tcp [12539.544278] Key type .llcrypt unregistered [12539.548645] Key type ._llcrypt unregistered [12565.850285] Key type ._llcrypt registered [12565.853162] Key type .llcrypt registered [12565.997324] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12567.326550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12567.488749] alg: No test for adler32 (adler32-zlib) [12568.816167] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [12569.190686] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12570.943276] Key type lgssc registered [12572.367207] Lustre: Echo OBD driver; http://www.lustre.org/ [12583.040647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12583.064314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12584.771286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12590.642055] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12595.173771] LustreError: 302728:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12595.186119] LustreError: 302728:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12600.299202] LustreError: 302729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12601.588349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12602.185960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12608.108865] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12617.931112] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12625.329596] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12633.563636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12634.069469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12641.968944] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12646.432851] LustreError: 304560:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12651.509157] LustreError: 304560:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.719629] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12658.987901] LustreError: 304563:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12659.005675] LustreError: 304563:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12669.183924] LustreError: 304560:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12669.205344] LustreError: 304560:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12679.619512] loop1: detected capacity change from 0 to 204800000 [12679.959617] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12679.990557] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12680.018883] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12685.294307] LustreError: 304563:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12685.329151] LustreError: 304563:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [12687.198361] loop1: detected capacity change from 0 to 204800000 [12694.888663] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12695.149774] Lustre: fs27215-OST0000: new disk, initializing [12695.155201] Lustre: srv-fs27215-OST0000: No data found on store. Initialize space. [12695.166468] Lustre: 306471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12695.279538] Lustre: fs27215-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12703.217499] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12713.954400] Lustre: server umount fs27215-OST0000 complete [12720.385632] LustreError: 305233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12720.427867] LustreError: 305233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [12726.253194] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12728.671543] loop1: detected capacity change from 0 to 204800000 [12731.390265] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12736.506894] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12736.888984] Lustre: fs27215-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12745.898211] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12758.496557] loop2: detected capacity change from 0 to 204800000 [12758.817708] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12758.836849] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12758.853668] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12765.449562] loop2: detected capacity change from 0 to 204800000 [12772.400918] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12772.608244] LustreError: fs27215-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12772.621491] LustreError: fs27215-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12772.637172] LustreError: 309514:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [12772.643673] LustreError: 309514:0:(tgt_mount.c:2128:server_put_super()) no obd fs27215-OST0001 [12772.653226] LustreError: 309514:0:(tgt_mount.c:116:server_deregister_mount()) fs27215-OST0001 not registered [12772.700721] Lustre: server umount fs27215-OST0001 complete [12772.704702] LustreError: 309514:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12784.915283] LustreError: 304562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12784.952314] LustreError: 304562:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [12787.895855] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12788.198936] Lustre: fs27215-OST0001: new disk, initializing [12788.208823] Lustre: srv-fs27215-OST0001: No data found on store. Initialize space. [12788.221354] Lustre: 310067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12788.372709] Lustre: fs27215-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12796.500560] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12807.927138] Lustre: server umount fs27215-OST0000 complete [12820.455468] Lustre: server umount fs27215-OST0001 complete [12837.466183] loop1: detected capacity change from 0 to 204800000 [12845.257614] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12845.708757] Lustre: fs27215-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12853.783205] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12862.005530] loop2: detected capacity change from 0 to 204800000 [12868.948450] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12876.955720] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12879.870710] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12884.974450] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12889.429656] Lustre: server umount fs27215-OST0000 complete [12900.199868] Lustre: server umount fs27215-OST0001 complete [12915.720456] LustreError: 308894:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12915.762444] LustreError: 308894:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 76 previous similar messages [12917.729983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12917.739353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12917.755592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12920.815565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12920.821319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12923.910810] Lustre: server umount lustre-OST0000 complete [12930.016631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12930.029699] LustreError: Skipped 1 previous similar message [12930.046528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12930.074405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12930.097874] Lustre: Skipped 1 previous similar message [12935.849951] Lustre: server umount lustre-MDT0000 complete [12941.391344] LustreError: 302709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075951 with bad export cookie 12389303872476157012 [12941.408434] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12941.435377] LustreError: 302709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12942.127898] Lustre: server umount lustre-MDT0001 complete [12957.500708] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12961.374399] Key type lgssc unregistered [12961.918774] LNet: 315994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12961.944440] LNetError: 315994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12961.967155] LNet: Removed LNI 192.168.204.146@tcp [12963.923773] Key type .llcrypt unregistered [12963.928923] Key type ._llcrypt unregistered [13006.585638] Key type ._llcrypt registered [13006.587178] Key type .llcrypt registered [13006.717861] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13026.719198] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13028.571225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13028.687418] alg: No test for adler32 (adler32-zlib) [13029.884926] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [13030.153973] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13031.880066] Key type lgssc registered [13033.710226] Lustre: Echo OBD driver; http://www.lustre.org/ [13084.282835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13084.345470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13085.879500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13085.962731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13086.100300] Lustre: lustre-MDT0000: new disk, initializing [13086.202475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13086.243785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13091.823511] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13104.498581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13104.582384] Lustre: 319852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13104.672692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13104.684812] Lustre: Skipped 1 previous similar message [13104.784381] Lustre: lustre-MDT0001: new disk, initializing [13104.916990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13105.001245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13105.028090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13111.020418] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13122.311312] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13130.529903] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13137.771246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13137.972810] Lustre: lustre-OST0000: new disk, initializing [13137.977731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13137.986526] Lustre: 321822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13138.048626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13145.800473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13145.814848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13146.275301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13147.827872] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13161.783837] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13164.373911] LustreError: 318024:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [13164.411865] Lustre: Mounted lustre-client - version 2.17.58_39_ga794951 [13165.717227] Lustre: Unmounted lustre-client [13179.879581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13179.918695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13179.929058] Lustre: Skipped 1 previous similar message [13183.259725] Lustre: server umount lustre-OST0000 complete [13190.118268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13190.138415] Lustre: Skipped 1 previous similar message [13190.150320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13190.161706] Lustre: Skipped 1 previous similar message [13194.869860] Lustre: server umount lustre-MDT0000 complete [13199.749754] LustreError: 319844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076209 with bad export cookie 6871596103889647676 [13199.759647] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13199.768238] LustreError: 319844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13200.643846] Lustre: server umount lustre-MDT0001 complete [13216.650242] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13220.572890] Key type lgssc unregistered [13225.055730] LNet: 323985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13225.066140] LNetError: 323985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13292.646078] LNet: Removed LNI 192.168.204.146@tcp [13293.789645] Key type .llcrypt unregistered [13293.800371] Key type ._llcrypt unregistered [13320.138892] Key type ._llcrypt registered [13320.141272] Key type .llcrypt registered [13320.264359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13340.167896] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13341.287100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13341.557477] alg: No test for adler32 (adler32-zlib) [13342.841752] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [13343.201541] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13345.039231] Key type lgssc registered [13346.439160] Lustre: Echo OBD driver; http://www.lustre.org/ [13385.350542] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 17:39:52 (1789076392) [13393.538753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13393.550771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13394.789439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13394.853905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13395.069052] Lustre: lustre-MDT0000: new disk, initializing [13395.280705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13395.317135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13400.845492] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13415.551922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13415.741346] Lustre: 327222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13415.836215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13415.841337] Lustre: Skipped 1 previous similar message [13415.988188] Lustre: lustre-MDT0001: new disk, initializing [13416.170291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13416.256678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13416.271170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13426.548443] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13441.297316] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13447.752470] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13456.843119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13457.076679] Lustre: lustre-OST0000: new disk, initializing [13457.080833] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13457.095240] Lustre: 329195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13457.343638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13463.102979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13463.126284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13463.267817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13464.532592] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13475.912255] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13492.206713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13492.238250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13492.247312] Lustre: Skipped 1 previous similar message [13497.313179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13497.325852] Lustre: Skipped 1 previous similar message [13498.223059] Lustre: server umount lustre-OST0000 complete [13503.456768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13503.467686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13503.473567] Lustre: Skipped 1 previous similar message [13503.479144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13507.564856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13507.570448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13512.681755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13512.691296] Lustre: Skipped 2 previous similar messages [13517.279989] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13517.634448] Lustre: server umount lustre-MDT0000 complete [13522.915504] LustreError: 327235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13522.958509] LustreError: 327235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13523.129071] LustreError: 327215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076532 with bad export cookie 11240266129699331347 [13523.145716] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13523.151050] LustreError: 327215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13523.657773] Lustre: server umount lustre-MDT0001 complete [13537.354569] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13540.771103] Key type lgssc unregistered [13541.428983] LNet: 331105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13541.436236] LNetError: 331105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13542.506925] LNet: Removed LNI 192.168.204.146@tcp [13543.753566] Key type .llcrypt unregistered [13543.757575] Key type ._llcrypt unregistered [13574.455782] Key type ._llcrypt registered [13574.457698] Key type .llcrypt registered [13574.626472] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13576.185895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13576.399661] alg: No test for adler32 (adler32-zlib) [13577.737864] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [13578.134147] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13579.967549] Key type lgssc registered [13581.369377] Lustre: Echo OBD driver; http://www.lustre.org/ [13593.457984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13593.496644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13595.391186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13600.419190] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13605.865619] LustreError: 332379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13605.885274] LustreError: 332379:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13610.628038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13611.098514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13616.323889] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13624.872522] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13632.872809] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13641.627313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13642.163968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13650.895959] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13660.879891] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13678.344828] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13680.742741] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13687.777078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13687.794966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13687.820936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13688.580876] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [13689.327162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13691.519225] Lustre: server umount lustre-MDT0000 complete [13693.700232] LustreError: 332374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13698.793658] LustreError: 334840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13698.826524] LustreError: 334840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13698.953233] LustreError: 332360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076708 with bad export cookie 8323343771254287990 [13698.961071] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13698.967358] LustreError: 332360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13699.553264] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13699.579802] Lustre: Skipped 1 previous similar message [13699.591597] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13699.595251] Lustre: Skipped 3 previous similar messages [13703.585533] LustreError: 332373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13703.587996] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13703.619052] LustreError: 332373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13708.718435] LustreError: 334840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13708.719552] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13708.746593] LustreError: 334840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13708.783735] Lustre: Skipped 1 previous similar message [13713.377555] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13713.809026] Lustre: server umount lustre-MDT0001 complete [13719.255954] Lustre: server umount lustre-OST0000 complete [13731.212519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13737.948605] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13741.137415] Lustre: 336588:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.146@tcp [13743.669438] Lustre: server umount MGS complete [13753.780922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13754.274943] LustreError: 337159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13754.290963] LustreError: 337159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13754.408859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13759.780993] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13770.491167] LustreError: 337156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13770.506596] LustreError: 337156:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [13771.739608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13772.287612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13778.083478] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13787.474994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13787.909522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13795.366881] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13805.884248] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13808.504758] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13817.469109] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13819.969748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13828.067326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13828.073872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13828.102880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13828.115247] Lustre: Skipped 1 previous similar message [13832.517968] Lustre: server umount lustre-OST0000 complete [13838.818666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13838.827700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13838.846731] Lustre: Skipped 1 previous similar message [13845.797872] Lustre: server umount lustre-MDT0000 complete [13846.510791] LustreError: 337159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13846.532080] LustreError: 337159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13850.007471] LustreError: 337139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076859 with bad export cookie 8323343771254288830 [13850.012895] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13850.025053] LustreError: 337139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13850.320364] Lustre: server umount lustre-MDT0001 complete [13864.245640] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13867.982587] Key type lgssc unregistered [13868.365513] LNet: 340596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13868.375173] LNetError: 340596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13868.393240] LNet: Removed LNI 192.168.204.146@tcp [13869.335687] Key type .llcrypt unregistered [13869.339408] Key type ._llcrypt unregistered [13894.102380] Key type ._llcrypt registered [13894.110013] Key type .llcrypt registered [13894.285485] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13913.072785] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13914.405743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13914.613726] alg: No test for adler32 (adler32-zlib) [13915.697281] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [13915.949632] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13917.607217] Key type lgssc registered [13919.104101] Lustre: Echo OBD driver; http://www.lustre.org/ [13963.916842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13963.958679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13965.174123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13965.235462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13965.375394] Lustre: lustre-MDT0000: new disk, initializing [13965.496984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13965.516299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13970.819634] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13985.921764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13986.148337] Lustre: 343760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13986.299714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13986.307464] Lustre: Skipped 1 previous similar message [13986.405242] Lustre: lustre-MDT0001: new disk, initializing [13986.574873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13986.642570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13986.674525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13992.794785] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14005.350228] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14013.784844] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14021.942844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14022.306453] Lustre: lustre-OST0000: new disk, initializing [14022.311428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14022.323344] Lustre: 345732:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14022.422833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14028.890405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14028.906492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14029.000873] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14029.079627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14041.294882] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14048.068320] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14048.495658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14053.984653] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14054.169468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14060.006668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14060.021754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14060.033237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14060.041257] Lustre: Skipped 1 previous similar message [14060.060933] Lustre: Skipped 1 previous similar message [14062.441563] Lustre: server umount lustre-OST0000 complete [14068.193596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14068.222195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14068.239935] Lustre: Skipped 1 previous similar message [14073.136340] Lustre: server umount lustre-MDT0000 complete [14077.461972] LustreError: 343751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789077087 with bad export cookie 7891151656502733788 [14077.466944] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14077.484834] LustreError: 343751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14077.731460] Lustre: server umount lustre-MDT0001 complete [14089.910900] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 17:51:36 (1789077096) [14117.522066] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [14129.371566] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [14171.219173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14171.525407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14171.556065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14171.671986] Lustre: lustre-MDT0000: new disk, initializing [14171.854334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14171.880694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14177.191492] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14188.417040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14188.532449] Lustre: 350493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14188.579836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14188.583867] Lustre: Skipped 1 previous similar message [14188.655369] Lustre: lustre-MDT0001: new disk, initializing [14188.742642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14188.782373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14188.799309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14193.544332] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14203.807712] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14209.979685] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14218.433861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14218.864180] Lustre: lustre-OST0000: new disk, initializing [14218.875530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14218.887423] Lustre: 352462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14219.064608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14226.003458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14226.012250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14226.131334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14226.891408] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14236.475448] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14282.721770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14282.753505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14282.756532] LustreError: Skipped 2 previous similar messages [14282.768710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14282.852867] Lustre: Skipped 2 previous similar messages [14287.846513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14287.871853] Lustre: Skipped 2 previous similar messages [14288.598736] Lustre: server umount lustre-OST0000 complete [14296.551198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14296.561906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14296.592812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14301.997082] Lustre: server umount lustre-MDT0000 complete [14303.217753] LustreError: 351718:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14303.227776] LustreError: 351718:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [14307.827782] LustreError: 350484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789077317 with bad export cookie 7891151656502734670 [14307.834741] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14307.844906] LustreError: 350484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14308.287771] Lustre: server umount lustre-MDT0001 complete [14323.902571] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14327.467298] Key type lgssc unregistered [14327.921965] LNet: 354641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14327.938333] LNetError: 354641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14327.963867] LNet: Removed LNI 192.168.204.146@tcp [14328.876478] Key type .llcrypt unregistered [14328.879259] Key type ._llcrypt unregistered [14438.735660] Key type ._llcrypt registered [14438.739685] Key type .llcrypt registered [14438.970213] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14439.422992] Key type .llcrypt unregistered [14439.426257] Key type ._llcrypt unregistered [14542.952740] Key type ._llcrypt registered [14542.957520] Key type .llcrypt registered [14543.030205] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14543.433548] Key type .llcrypt unregistered [14543.435956] Key type ._llcrypt unregistered [14583.126133] Key type ._llcrypt registered [14583.133305] Key type .llcrypt registered [14583.349444] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14583.867506] Key type .llcrypt unregistered [14583.875875] Key type ._llcrypt unregistered [14626.289084] Key type ._llcrypt registered [14626.290835] Key type .llcrypt registered [14626.459852] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14626.813546] Key type .llcrypt unregistered [14626.815946] Key type ._llcrypt unregistered [14724.418371] Key type ._llcrypt registered [14724.422790] Key type .llcrypt registered [14724.527402] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14724.845301] Key type .llcrypt unregistered [14724.848136] Key type ._llcrypt unregistered [14763.713457] Key type ._llcrypt registered [14763.718420] Key type .llcrypt registered [14763.881717] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14764.190579] Key type .llcrypt unregistered [14764.195876] Key type ._llcrypt unregistered [14801.806887] Key type ._llcrypt registered [14801.829851] Key type .llcrypt registered [14802.058400] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14802.416367] Key type .llcrypt unregistered [14802.421571] Key type ._llcrypt unregistered [14841.586826] Key type ._llcrypt registered [14841.591336] Key type .llcrypt registered [14841.758386] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14842.153502] Key type .llcrypt unregistered [14842.159280] Key type ._llcrypt unregistered [14881.481957] Key type ._llcrypt registered [14881.484773] Key type .llcrypt registered [14881.588586] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14881.943348] Key type .llcrypt unregistered [14881.945692] Key type ._llcrypt unregistered [14922.046416] Key type ._llcrypt registered [14922.048568] Key type .llcrypt registered [14922.219806] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14922.586347] Key type .llcrypt unregistered [14922.588125] Key type ._llcrypt unregistered [14996.339425] Key type ._llcrypt registered [14996.342440] Key type .llcrypt registered [14996.468911] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14996.752327] Key type .llcrypt unregistered [14996.755905] Key type ._llcrypt unregistered [15030.568402] Key type ._llcrypt registered [15030.571362] Key type .llcrypt registered [15030.787824] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [15031.081327] Key type .llcrypt unregistered [15031.087372] Key type ._llcrypt unregistered [15061.753215] Key type ._llcrypt registered [15061.754860] Key type .llcrypt registered [15061.958372] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [15062.372312] Key type .llcrypt unregistered [15062.378108] Key type ._llcrypt unregistered [15132.267587] Key type ._llcrypt registered [15132.270662] Key type .llcrypt registered [15132.391692] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [15150.059382] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [15150.877515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15150.921106] alg: No test for adler32 (adler32-zlib) [15152.058066] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [15152.424797] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [15154.191262] Key type lgssc registered [15155.403227] Lustre: Echo OBD driver; http://www.lustre.org/ [15195.700152] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15195.712495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15196.957283] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [15196.992756] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [15197.039891] Lustre: tst250fs-MDT0000: new disk, initializing [15197.112164] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15197.179635] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15203.086625] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15214.642834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15214.782383] Lustre: 374990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15214.851778] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [15214.857573] Lustre: Skipped 1 previous similar message [15214.951461] Lustre: tst250fs-MDT0001: new disk, initializing [15215.081238] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15215.134738] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15215.147648] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15220.369786] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15225.464447] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [15232.946923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15233.384298] Lustre: tst250fs-OST0000: new disk, initializing [15233.387151] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [15233.394203] Lustre: 376626:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15233.488657] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15240.825928] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15240.840860] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15240.898115] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15240.992173] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15252.333278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15252.562760] Lustre: tst250fs-OST0001: new disk, initializing [15252.568152] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [15252.579316] Lustre: 377495:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15252.683654] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15260.208399] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15260.289027] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15260.300147] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15260.450024] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15266.595739] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [15269.296056] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [15269.306440] Lustre: Skipped 2 previous similar messages [15291.365348] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15291.371241] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15291.394458] Lustre: Skipped 1 previous similar message [15295.250987] Lustre: server umount tst250fs-MDT0000 complete [15299.540468] LustreError: 374982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789078309 with bad export cookie 8929025105852352386 [15299.543496] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15299.565881] LustreError: 374982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15300.178583] Lustre: server umount tst250fs-MDT0001 complete [15315.052616] Lustre: server umount tst250fs-OST0000 complete [15317.985365] Lustre: 373212:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789078311/real 1789078311] req@ffff962df1674a80 x1875984422867712/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789078327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15318.026487] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15318.053329] Lustre: Skipped 2 previous similar messages [15320.562871] Lustre: server umount tst250fs-OST0001 complete [15346.180427] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [15356.309917] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [15400.693101] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 18:13:28 (1789078408) [15402.459820] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [15419.018109] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [15430.350828] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [15466.432837] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15147 sec ========= 18:14:33 (1789078473) [15468.417018] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:14:35 (1789078475) === [15472.559959] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:14:39 (1789078479) === [15526.125820] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [15530.252103] Key type lgssc unregistered [15530.664966] LNet: 386626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15530.683327] LNetError: 386626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15530.718860] LNet: Removed LNI 192.168.204.146@tcp [15531.890183] Key type .llcrypt unregistered [15531.891807] Key type ._llcrypt unregistered