[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 617224061 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003127] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008585] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011323] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014072] *** VALIDATE selinux *** [ 0.024409] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030058] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033104] *** VALIDATE tmpfs *** [ 0.034525] *** VALIDATE proc *** [ 0.035265] *** VALIDATE cgroup *** [ 0.036016] *** VALIDATE cgroup2 *** [ 0.037304] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044081] debug: unmapping init [mem 0xffffffffb6259000-0xffffffffb6260fff] [ 0.046402] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047968] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049026] ... version: 2 [ 0.050022] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053052] ... max period: 00007fffffffffff [ 0.054020] ... fixed-purpose events: 3 [ 0.055018] ... event mask: 000000070000000f [ 0.056347] rcu: Hierarchical SRCU implementation. [ 0.058701] smp: Bringing up secondary CPUs ... [ 0.059706] x86: Booting SMP configuration: [ 0.060031] .... node #0, CPUs: #1 #2 #3 [ 0.071289] smp: Brought up 1 node, 4 CPUs [ 0.073017] smpboot: Max logical packages: 1 [ 0.074018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.204904] node 0 deferred pages initialised in 127ms [ 0.212117] devtmpfs: initialized [ 0.213758] x86/mm: Memory block size: 128MB [ 0.218000] gcov: version magic: 0x41383552 [ 0.221328] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.222108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.232132] pinctrl core: initialized pinctrl subsystem [ 0.241485] [ 0.242008] ************************************************************* [ 0.251092] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.255014] ** ** [ 0.259014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.262021] ** ** [ 0.265015] ** This means that this kernel is built to expose internal ** [ 0.268012] ** IOMMU data structures, which may compromise security on ** [ 0.272160] ** your system. ** [ 0.276017] ** ** [ 0.280020] ** If you see this message and you are not debugging the ** [ 0.284018] ** kernel, report this immediately to your vendor! ** [ 0.289129] ** ** [ 0.293016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.297019] ************************************************************* [ 0.298000] NET: Registered protocol family 16 [ 0.300865] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.301000] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.303080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.310175] cpuidle: using governor menu [ 0.315371] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.319816] PCI: Using configuration type 1 for base access [ 0.322153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.336210] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.337028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.339220] cryptd: max_cpu_qlen set to 1000 [ 0.343000] ACPI: Added _OSI(Module Device) [ 0.344017] ACPI: Added _OSI(Processor Device) [ 0.346014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.348016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.352598] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.360545] ACPI: Interpreter enabled [ 0.362076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.364013] ACPI: Using IOAPIC for interrupt routing [ 0.366166] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.369479] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.379223] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.381049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.383086] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.386104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.391800] acpiphp: Slot [2] registered [ 0.394188] acpiphp: Slot [5] registered [ 0.395250] acpiphp: Slot [6] registered [ 0.397155] acpiphp: Slot [7] registered [ 0.399194] acpiphp: Slot [8] registered [ 0.400150] acpiphp: Slot [9] registered [ 0.402156] acpiphp: Slot [10] registered [ 0.403000] acpiphp: Slot [3] registered [ 0.403000] acpiphp: Slot [4] registered [ 0.406149] acpiphp: Slot [11] registered [ 0.407195] acpiphp: Slot [12] registered [ 0.409103] acpiphp: Slot [13] registered [ 0.410104] acpiphp: Slot [14] registered [ 0.411103] acpiphp: Slot [15] registered [ 0.413102] acpiphp: Slot [16] registered [ 0.415115] acpiphp: Slot [17] registered [ 0.416168] acpiphp: Slot [18] registered [ 0.418180] acpiphp: Slot [19] registered [ 0.420105] acpiphp: Slot [20] registered [ 0.422113] acpiphp: Slot [21] registered [ 0.423161] acpiphp: Slot [22] registered [ 0.425109] acpiphp: Slot [23] registered [ 0.427188] acpiphp: Slot [24] registered [ 0.428156] acpiphp: Slot [25] registered [ 0.430119] acpiphp: Slot [26] registered [ 0.432107] acpiphp: Slot [27] registered [ 0.434105] acpiphp: Slot [28] registered [ 0.435182] acpiphp: Slot [29] registered [ 0.437148] acpiphp: Slot [30] registered [ 0.439125] acpiphp: Slot [31] registered [ 0.440068] PCI host bridge to bus 0000:00 [ 0.442025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.445023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.447025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.450025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.453029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.457034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.459202] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.462203] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.466667] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.477017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.484664] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.487028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.489089] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.492020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.495647] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.499020] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.502053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.506160] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.513018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.527021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.532010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.541181] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.550020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.558023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.581023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.593122] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.601021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.608022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.629020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.644673] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.653021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.661021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.678020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.691950] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.702019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.714021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.730016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.743104] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.759031] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.771027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.795020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.808000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.818020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.825020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.841021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.855908] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.858505] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.861393] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.863410] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.866238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.870116] iommu: Default domain type: Passthrough [ 0.875428] SCSI subsystem initialized [ 0.876189] ACPI: bus type USB registered [ 0.881186] usbcore: registered new interface driver usbfs [ 0.884119] usbcore: registered new interface driver hub [ 0.886111] usbcore: registered new device driver usb [ 0.888402] pps_core: LinuxPPS API ver. 1 registered [ 0.891015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.895084] PTP clock support registered [ 0.899251] EDAC MC: Ver: 3.0.0 [ 0.903212] PCI: Using ACPI for IRQ routing [ 0.906155] NetLabel: Initializing [ 0.907018] NetLabel: domain hash size = 128 [ 0.909012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.912099] NetLabel: unlabeled traffic allowed by default [ 0.914414] vgaarb: loaded [ 0.916110] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.918014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.925914] clocksource: Switched to clocksource kvm-clock [ 1.134662] VFS: Disk quotas dquot_6.6.0 [ 1.137605] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.140731] *** VALIDATE ramfs *** [ 1.142294] *** VALIDATE hugetlbfs *** [ 1.144345] pnp: PnP ACPI init [ 1.147173] pnp: PnP ACPI: found 6 devices [ 1.163169] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.167117] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.169857] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.172244] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.175185] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.177824] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.180989] NET: Registered protocol family 2 [ 1.183793] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.189939] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.193816] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.199744] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.203593] TCP: Hash tables configured (established 65536 bind 65536) [ 1.206933] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.210836] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.213981] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.220558] NET: Registered protocol family 1 [ 1.223565] RPC: Registered named UNIX socket transport module. [ 1.225638] RPC: Registered udp transport module. [ 1.227066] RPC: Registered tcp transport module. [ 1.228923] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.231696] NET: Registered protocol family 44 [ 1.233078] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.235560] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.237852] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.240709] PCI: CLS 0 bytes, default 64 [ 1.242922] Unpacking initramfs... [ 3.007847] debug: unmapping init [mem 0xffff9cc97cc54000-0xffff9cc97ffbffff] [ 3.012303] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.015281] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.018628] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.675402] Initialise system trusted keyrings [ 3.677245] Key type blacklist registered [ 3.679521] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.689881] zbud: loaded [ 3.693231] *** VALIDATE nfs *** [ 3.694657] *** VALIDATE nfs4 *** [ 3.696684] pstore: using deflate compression [ 3.701401] Platform Keyring initialized [ 3.825593] NET: Registered protocol family 38 [ 3.827752] Key type asymmetric registered [ 3.829812] Asymmetric key parser 'x509' registered [ 3.832465] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.836556] io scheduler mq-deadline registered [ 3.839074] io scheduler kyber registered [ 3.841588] io scheduler bfq registered [ 3.843962] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.848422] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.852298] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.855852] ACPI: Power Button [PWRF] [ 3.862567] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.870195] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.890962] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.904373] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.939871] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.969075] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.996153] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.003342] Non-volatile memory driver v1.3 [ 4.005338] Linux agpgart interface v0.103 [ 4.052985] virtio_blk virtio1: [vda] 139528 512-byte logical blocks (71.4 MB/68.1 MiB) [ 4.056306] vda: detected capacity change from 0 to 71438336 [ 4.109766] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.113528] vdb: detected capacity change from 0 to 1073741824 [ 4.134745] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.138356] vdc: detected capacity change from 0 to 2621440000 [ 4.162705] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.166129] vdd: detected capacity change from 0 to 2621440000 [ 4.186386] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.189738] vde: detected capacity change from 0 to 4294967296 [ 4.220994] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.228955] vdf: detected capacity change from 0 to 4294967296 [ 4.245649] libphy: Fixed MDIO Bus: probed [ 4.258242] usbcore: registered new interface driver usbserial_generic [ 4.263398] usbserial: USB Serial support registered for generic [ 4.268629] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.276658] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.278859] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.281635] mousedev: PS/2 mouse device common for all mice [ 4.285364] rtc_cmos 00:05: RTC can wake from S4 [ 4.290386] rtc_cmos 00:05: registered as rtc0 [ 4.292695] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.297680] intel_pstate: CPU model not supported [ 4.299309] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.309571] hid: raw HID events driver (C) Jiri Kosina [ 4.312633] usbcore: registered new interface driver usbhid [ 4.314785] usbhid: USB HID core driver [ 4.317189] drop_monitor: Initializing network drop monitor service [ 4.319942] Initializing XFRM netlink socket [ 4.321213] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.328561] NET: Registered protocol family 10 [ 4.336535] Segment Routing with IPv6 [ 4.338661] NET: Registered protocol family 17 [ 4.339273] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.341383] mpls_gso: MPLS GSO support [ 4.353979] RAS: Correctable Errors collector initialized. [ 4.356684] AVX version of gcm_enc/dec engaged. [ 4.358853] AES CTR mode by8 optimization enabled [ 4.453578] sched_clock: Marking stable (4453555267, 0)->(5660383392, -1206828125) [ 4.461368] registered taskstats version 1 [ 4.463737] Loading compiled-in X.509 certificates [ 4.466258] zswap: loaded using pool lzo/zbud [ 4.498865] Key type big_key registered [ 4.515814] Key type encrypted registered [ 4.517749] ima: No TPM chip found, activating TPM-bypass! [ 4.520565] ima: Allocated hash algorithm: sha1 [ 4.522697] ima: No architecture policies found [ 4.524901] evm: Initialising EVM extended attributes: [ 4.527451] evm: security.selinux [ 4.528984] evm: security.ima [ 4.530489] evm: security.capability [ 4.531950] evm: HMAC attrs: 0x1 [ 4.534756] rtc_cmos 00:05: setting system clock to 2026-06-09 17:53:09 UTC (1781027589) [ 4.542451] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 4.546169] debug: unmapping init [mem 0xffffffffb5f82000-0xffffffffb6258fff] [ 4.555265] Write protecting the kernel read-only data: 28672k [ 4.560383] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 4.563828] debug: unmapping init [mem 0xffffffffb4f14000-0xffffffffb4ffffff] [ 4.606477] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.616547] systemd[1]: Detected virtualization kvm. [ 4.618821] systemd[1]: Detected architecture x86-64. [ 4.621289] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.652365] systemd[1]: No hostname configured. [ 4.654429] systemd[1]: Set hostname to . [ 4.656805] random: systemd: uninitialized urandom read (16 bytes read) [ 4.659579] systemd[1]: Initializing machine ID from random generator. [ 4.966431] random: systemd: uninitialized urandom read (16 bytes read) [ 4.969873] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.984531] random: systemd: uninitialized urandom read (16 bytes read) [ 4.990513] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.000784] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.212491] device-mapper: uevent: version 1.0.3 [ 6.215617] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.595512] random: fast init done [ 7.653149] virtio_net virtio0 ens2: renamed from eth0 [ 7.853245] scsi host0: ata_piix [ 7.929923] scsi host1: ata_piix [ 7.933034] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.940791] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.579946] hrtimer: interrupt took 8062563 ns [ 14.248909] random: crng init done [ 14.250415] random: 7 urandom warning(s) missed due to ratelimiting [ 15.056879] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.910447] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.036369] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.446899] SELinux: Disabled at runtime. [ 18.516994] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.525863] systemd[1]: Detected virtualization kvm. [ 18.527746] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.225596] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.229486] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.236374] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.242286] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.247141] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.254367] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.265475] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Reached targ[ 19.451832] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started udev Coldplug all Devices. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 20.382692] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.187621] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.249613] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.771172] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.914573] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit)[ 26.180824] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit)[ 26.874680] NFS: Registering the id_resolver key type [ 26.880506] Key type id_resolver registered [ 26.882011] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg350-server login: [ 87.792242] libcfs: loading out-of-tree module taints kernel. [ 87.842752] Key type ._llcrypt registered [ 87.845298] Key type .llcrypt registered [ 87.933857] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 104.660769] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 106.123341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 106.136262] alg: No test for adler32 (adler32-zlib) [ 107.587990] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 108.234477] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 109.960251] Key type lgssc registered [ 111.810159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.346502] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 163.725546] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 177.076697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 177.111750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.374660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.410746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.530985] Lustre: lustre-MDT0000: new disk, initializing [ 178.612792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.630606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.527604] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.632643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.763772] Lustre: 6405:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.792458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.796825] Lustre: Skipped 1 previous similar message [ 195.901158] Lustre: lustre-MDT0001: new disk, initializing [ 196.003883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.035499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 196.049402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 199.724294] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.060876] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 213.842243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.068358] Lustre: lustre-OST0000: new disk, initializing [ 214.072273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 214.077447] Lustre: 8404:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.144324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.119686] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.182973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.196345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.273399] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 233.866087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.984767] Lustre: lustre-OST0001: new disk, initializing [ 233.988827] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 234.000915] Lustre: 9508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.063789] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.551767] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.699113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 240.708445] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 240.787579] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 252.972797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.425613] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 267.517389] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing check_logdir /tmp/testlogs/ [ 271.808455] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing yml_node [ 275.997773] Lustre: DEBUG MARKER: Client: 2.17.54.1 [ 278.490532] Lustre: DEBUG MARKER: MDS: 2.17.54.1 [ 281.308128] Lustre: DEBUG MARKER: OSS: 2.17.54.1 [ 282.930205] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 9 13:57:45 EDT 2026 [ 296.273340] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 297.528836] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.802300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.805377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.820564] Lustre: Skipped 2 previous similar messages [ 312.830640] Lustre: Skipped 3 previous similar messages [ 317.920306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.931465] Lustre: Skipped 3 previous similar messages [ 318.395302] Lustre: server umount lustre-MDT0000 complete [ 321.839521] LustreError: 6399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781027906 with bad export cookie 18138866110744642683 [ 321.852149] LustreError: 6399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 321.852675] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 323.043862] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.054068] Lustre: Skipped 1 previous similar message [ 323.061865] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.072851] Lustre: Skipped 1 previous similar message [ 326.049209] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 331.168753] LustreError: 8410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.170375] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 331.190076] LustreError: 8410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 331.203238] Lustre: Skipped 2 previous similar messages [ 336.351244] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 336.351941] LustreError: 8410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.378477] LustreError: 8410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 336.610460] Lustre: server umount lustre-MDT0001 complete [ 340.307652] Lustre: server umount lustre-OST0000 complete [ 343.841360] Lustre: server umount lustre-OST0001 complete [ 349.454916] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 355.953981] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 382.302703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 382.506527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 382.527710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 382.575975] Lustre: lustre-MDT0000: new disk, initializing [ 382.624794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.642890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 385.990506] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 396.301791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.400615] Lustre: 14927:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 396.455118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 396.462823] Lustre: Skipped 1 previous similar message [ 396.533517] Lustre: lustre-MDT0001: new disk, initializing [ 396.633889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.671916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.677449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 400.329574] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 407.069622] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 409.736804] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 414.390160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.589322] Lustre: lustre-OST0000: new disk, initializing [ 414.593554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 414.600894] Lustre: 16764:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 414.679139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.919056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 415.931628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 416.040848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 418.799354] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 424.464611] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 427.949809] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 428.103646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 431.091277] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 431.237722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 436.709391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.710355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 436.715828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 436.715836] Lustre: Skipped 2 previous similar messages [ 436.717985] Lustre: Skipped 2 previous similar messages [ 438.628915] Lustre: server umount lustre-OST0000 complete [ 441.825866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.836787] Lustre: Skipped 1 previous similar message [ 447.462296] Lustre: server umount lustre-MDT0000 complete [ 449.718836] LustreError: 15753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028034 with bad export cookie 18138866110744644223 [ 449.725492] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 449.728962] LustreError: 15753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 449.935246] Lustre: server umount lustre-MDT0001 complete [ 453.452897] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:00:36 (1781028036) [ 465.785850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.086113] LustreError: 19538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.142310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.936222] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 471.520691] LustreError: 19539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.411977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.417532] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 484.112424] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 486.450716] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 487.782470] Lustre: Failing over lustre-MDT0000 [ 487.936949] Lustre: server umount lustre-MDT0000 complete [ 490.977699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.977793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 490.990221] LustreError: 19533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.996620] LustreError: Skipped 1 previous similar message [ 491.007245] LustreError: 19533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 500.194829] LustreError: 19533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.206614] LustreError: 19533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 504.260643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.347333] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 508.282623] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 509.923427] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 509.936462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 509.959473] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 514.991246] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 519.137809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.146275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 519.149645] Lustre: Skipped 2 previous similar messages [ 519.160761] Lustre: Skipped 6 previous similar messages [ 524.445126] Lustre: server umount lustre-MDT0000 complete [ 525.282210] LustreError: 19538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.299160] LustreError: 19538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 527.427053] LustreError: 19520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028112 with bad export cookie 18138866110744645637 [ 527.434972] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.437914] LustreError: 19520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 527.729527] Lustre: server umount lustre-MDT0001 complete [ 534.195320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.548997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 534.555696] Lustre: Skipped 2 previous similar messages [ 537.893796] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 544.934581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.404671] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 553.625969] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 556.555208] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 558.038097] Lustre: Failing over lustre-MDT0000 [ 558.202730] Lustre: server umount lustre-MDT0000 complete [ 560.608778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.615399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 560.640584] LustreError: 22687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.652739] LustreError: 22687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 575.298691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.402528] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.894498] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 581.105360] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 581.106135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 581.121228] Lustre: Skipped 1 previous similar message [ 581.140662] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 585.389292] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 590.309204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 590.313233] Lustre: Skipped 3 previous similar messages [ 594.560817] Lustre: server umount lustre-MDT0000 complete [ 596.853507] LustreError: 23456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028181 with bad export cookie 18138866110744646575 [ 596.859156] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.867111] LustreError: 23456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 597.126724] Lustre: server umount lustre-MDT0001 complete [ 602.302382] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:03:05 (1781028185) [ 613.287221] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 618.970252] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 642.334373] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 648.292041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.428393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 648.447903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 648.496233] Lustre: lustre-MDT0000: new disk, initializing [ 648.546863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 650.655601] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 657.756237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.837991] Lustre: 28806:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 658.042542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 660.386219] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 665.834542] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 668.110474] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 672.596859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 672.767590] Lustre: 30646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 672.827319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 672.835149] Lustre: Skipped 4 previous similar messages [ 674.707144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 674.739192] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 674.794638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 674.813784] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 676.591488] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 682.361705] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 688.341926] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 705.503942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 705.507175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 705.510682] LustreError: Skipped 1 previous similar message [ 705.511521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 705.522950] Lustre: Skipped 4 previous similar messages [ 705.534476] Lustre: Skipped 3 previous similar messages [ 715.231141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 715.385567] Lustre: server umount lustre-OST0000 complete [ 719.331734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 724.254350] Lustre: server umount lustre-MDT0000 complete [ 725.986147] LustreError: 29641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 725.995761] LustreError: 29641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 726.729600] LustreError: 28799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028311 with bad export cookie 18138866110744646981 [ 726.736796] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 726.743823] LustreError: 28799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 727.018772] Lustre: server umount lustre-MDT0001 complete [ 733.410934] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 734.922306] Key type lgssc unregistered [ 735.088543] LNet: 32539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.091485] LNetError: 32539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.104801] LNet: Removed LNI 192.168.203.150@tcp [ 735.648158] Key type .llcrypt unregistered [ 735.650190] Key type ._llcrypt unregistered [ 751.879197] Key type ._llcrypt registered [ 751.881469] Key type .llcrypt registered [ 751.951555] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 752.615462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 752.630871] alg: No test for adler32 (adler32-zlib) [ 753.585314] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 753.720890] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 755.359156] Key type lgssc registered [ 756.149581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 761.567670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 761.590161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.964107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 765.970116] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 772.074982] LustreError: 33914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 772.089789] LustreError: 33914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 772.840280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.086809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 775.852351] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 780.621217] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 783.256768] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 787.800873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 788.068726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.728939] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 796.041980] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 797.305564] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 819.645443] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:06:42 (1781028402) [ 820.904458] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 822.063164] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 825.941637] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:06:49 (1781028409) [ 830.212586] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:06:53 (1781028413) [ 830.672804] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 834.967498] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 834.974978] Lustre: Skipped 2 previous similar messages [ 835.919393] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:06:59 (1781028419) [ 840.628869] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 840.635699] Lustre: Skipped 4 previous similar messages [ 848.798384] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 848.800832] Lustre: Skipped 8 previous similar messages [ 849.660224] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:07:12 (1781028432) [ 851.313647] Lustre: *** cfs_fail_loc=131b, val=2*** [ 853.637482] Lustre: *** cfs_fail_loc=131b, val=2*** [ 858.781916] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:07:21 (1781028441) [ 867.246472] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 867.251240] Lustre: Skipped 4 previous similar messages [ 868.380832] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:07:31 (1781028451) [ 879.585582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 879.589061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.593029] LustreError: Skipped 1 previous similar message [ 879.609964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 879.615185] Lustre: Skipped 1 previous similar message [ 884.706374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.709824] Lustre: Skipped 1 previous similar message [ 889.825671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.829124] Lustre: Skipped 1 previous similar message [ 891.359132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 891.470422] Lustre: server umount lustre-OST0000 complete [ 894.945651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 894.949703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 894.963232] Lustre: Skipped 2 previous similar messages [ 894.976190] Lustre: Skipped 1 previous similar message [ 900.064714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 900.069970] Lustre: Skipped 1 previous similar message [ 908.255120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 908.387700] Lustre: server umount lustre-MDT0000 complete [ 910.305212] LustreError: 33915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 910.318201] LustreError: 33915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 910.777589] LustreError: 36664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028495 with bad export cookie 15915058135680612738 [ 910.782503] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 910.796485] LustreError: 36664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 911.025663] Lustre: server umount lustre-MDT0001 complete [ 917.977359] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 919.636521] Key type lgssc unregistered [ 919.831800] LNet: 41571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.837848] LNetError: 41571:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.851760] LNet: Removed LNI 192.168.203.150@tcp [ 920.359712] Key type .llcrypt unregistered [ 920.362186] Key type ._llcrypt unregistered [ 930.088925] Key type ._llcrypt registered [ 930.090591] Key type .llcrypt registered [ 930.185723] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 939.136376] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 939.620395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 939.630851] alg: No test for adler32 (adler32-zlib) [ 940.526796] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 940.638528] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 942.239168] Key type lgssc registered [ 942.853692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 963.126239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 963.141859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.320621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 964.355403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 964.446108] Lustre: lustre-MDT0000: new disk, initializing [ 964.516224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.533213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 967.210508] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 974.199608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.263717] Lustre: 44668:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 974.299775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 974.302825] Lustre: Skipped 1 previous similar message [ 974.351501] Lustre: lustre-MDT0001: new disk, initializing [ 974.403193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 974.426573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 974.433459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 976.630134] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 981.490669] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 983.551884] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 988.014975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 988.206798] Lustre: lustre-OST0000: new disk, initializing [ 988.210381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 988.216527] Lustre: 46502:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 988.275218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.708885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 988.715265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 988.810192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 991.973541] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 997.107230] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 999.419333] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 999.536209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1002.072073] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1002.192286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1004.006716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.017099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.020170] Lustre: Skipped 1 previous similar message [ 1007.012633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1007.016698] Lustre: Skipped 1 previous similar message [ 1009.394455] Lustre: server umount lustre-OST0000 complete [ 1012.192536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.193913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1012.203798] Lustre: Skipped 2 previous similar messages [ 1012.210314] Lustre: Skipped 1 previous similar message [ 1017.313381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.316273] Lustre: Skipped 1 previous similar message [ 1017.824991] Lustre: server umount lustre-MDT0000 complete [ 1019.839505] LustreError: 44660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028604 with bad export cookie 12661041974869052454 [ 1019.844542] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1019.849510] LustreError: 44660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1020.022795] Lustre: server umount lustre-MDT0001 complete [ 1025.138284] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:10:08 (1781028608) [ 1028.981563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.256931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.418531] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1034.721141] LustreError: 48861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.734288] LustreError: 48861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1036.309801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.503247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1038.579084] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1041.952806] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1044.047801] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1048.462502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1048.719977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.827215] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1055.104394] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1067.791637] LustreError: 51212:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1068.256767] Lustre: Modifying parameter general.timeout=1 in log params [ 1069.565906] Lustre: Modifying parameter general.timeout=4 in log params [ 1069.568834] Lustre: Skipped 2 previous similar messages [ 1071.880043] Lustre: Modifying parameter general.timeout=9 in log params [ 1071.885889] Lustre: Skipped 4 previous similar messages [ 1075.952858] Lustre: Modifying parameter general.timeout=18 in log params [ 1075.955485] Lustre: Skipped 8 previous similar messages [ 1084.100895] Lustre: Modifying parameter general.timeout=35 in log params [ 1084.104457] Lustre: Skipped 16 previous similar messages [ 1100.394834] Lustre: Modifying parameter general.timeout=70 in log params [ 1100.400519] Lustre: Skipped 34 previous similar messages [ 1135.501636] Lustre: Disabling parameter general.timeout= in log params [ 1135.505508] Lustre: Skipped 59 previous similar messages [ 1137.075141] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:12:00 (1781028720) [ 1138.391923] Lustre: 58127:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1140.068568] Lustre: 58324:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1140.076677] Lustre: 58324:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 1142.800683] Lustre: 58634:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1142.806900] Lustre: 58634:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 1155.187048] Lustre: 59989:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1155.203551] LustreError: 59989:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1157.431745] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:12:20 (1781028740) [ 1163.745817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.747606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.754074] Lustre: Skipped 2 previous similar messages [ 1163.760680] Lustre: Skipped 1 previous similar message [ 1169.896683] Lustre: server umount lustre-MDT0000 complete [ 1171.634124] LustreError: 51262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028756 with bad export cookie 12661041974869053336 [ 1171.637553] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.640915] LustreError: 51262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1171.803221] Lustre: server umount lustre-MDT0001 complete [ 1183.909540] Lustre: server umount lustre-OST0000 complete [ 1197.755400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1197.900684] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1198.081604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.388612] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1205.717481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.774372] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1205.792215] Lustre: 62514:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1208.002571] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1211.047223] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1212.747334] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1216.286484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1216.385907] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1219.233996] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1222.210165] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1228.430325] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1228.437067] Lustre: Skipped 23 previous similar messages [ 1228.457870] Lustre: 64882:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1231.329076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1231.329886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.332776] LustreError: Skipped 1 previous similar message [ 1231.338166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1231.342786] Lustre: Skipped 4 previous similar messages [ 1233.393160] Lustre: server umount lustre-OST0000 complete [ 1236.449965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.457056] Lustre: Skipped 2 previous similar messages [ 1237.371686] Lustre: server umount lustre-MDT0000 complete [ 1239.422966] LustreError: 62504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028824 with bad export cookie 12661041974869055870 [ 1239.425915] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1239.429356] LustreError: 62504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1239.624725] Lustre: server umount lustre-MDT0001 complete [ 1244.794513] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1246.032724] Key type lgssc unregistered [ 1246.191475] LNet: 66065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.198861] LNetError: 66065:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.209593] LNet: Removed LNI 192.168.203.150@tcp [ 1246.604247] Key type .llcrypt unregistered [ 1246.605907] Key type ._llcrypt unregistered [ 1258.592277] Key type ._llcrypt registered [ 1258.595405] Key type .llcrypt registered [ 1258.648203] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1259.246768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1259.263116] alg: No test for adler32 (adler32-zlib) [ 1260.168857] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 1260.304966] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1261.919164] Key type lgssc registered [ 1262.418651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1265.982126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1265.992362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.265216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.352395] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1274.781512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.079223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1276.978373] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1280.255978] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1281.915141] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1285.030029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1285.292937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.310814] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1291.175888] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1304.930091] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1310.178108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.182631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1310.184790] Lustre: Skipped 1 previous similar message [ 1310.190172] Lustre: Skipped 2 previous similar messages [ 1314.398869] Lustre: server umount lustre-MDT0000 complete [ 1315.296617] LustreError: 67441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.302903] LustreError: 67441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1315.974874] LustreError: 69952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028900 with bad export cookie 1725267947709572978 [ 1315.978761] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1315.982107] LustreError: 69952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1316.124823] Lustre: server umount lustre-MDT0001 complete [ 1327.204363] Lustre: server umount lustre-OST0000 complete [ 1340.325948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1340.471416] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1340.488768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1340.608909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1342.584228] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1347.285963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.348595] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1347.364319] Lustre: 72137:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1347.594153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.592668] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1353.054697] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1354.916429] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1358.315125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1358.448079] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1358.659328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.563259] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1364.944068] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1365.918084] Lustre: 74515:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1366.943595] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1369.060871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.060929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1369.068429] Lustre: Skipped 1 previous similar message [ 1369.076716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1369.080387] Lustre: Skipped 1 previous similar message [ 1374.177035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1374.178889] Lustre: Skipped 1 previous similar message [ 1374.956726] Lustre: server umount lustre-OST0000 complete [ 1378.271972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1378.276261] LustreError: Skipped 1 previous similar message [ 1378.283665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.290294] Lustre: Skipped 1 previous similar message [ 1378.296972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.814962] Lustre: server umount lustre-MDT0000 complete [ 1384.261100] LustreError: 72128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028969 with bad export cookie 1725267947709573790 [ 1384.263886] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1384.265719] LustreError: 72128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1384.388264] Lustre: server umount lustre-MDT0001 complete [ 1388.725232] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1390.118362] Key type lgssc unregistered [ 1390.304102] LNet: 75703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.308291] LNetError: 75703:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.322483] LNet: Removed LNI 192.168.203.150@tcp [ 1390.735210] Key type .llcrypt unregistered [ 1390.738270] Key type ._llcrypt unregistered [ 1402.877483] Key type ._llcrypt registered [ 1402.879309] Key type .llcrypt registered [ 1402.931710] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1403.317167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1403.325873] alg: No test for adler32 (adler32-zlib) [ 1404.212624] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 1404.361560] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1405.983902] Key type lgssc registered [ 1406.456813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1410.243887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1410.258088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.502055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.105688] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1417.476883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.752948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1419.421351] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1422.641319] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1424.195137] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1427.051169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1427.301042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.582963] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1432.384573] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1441.082249] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1441.878969] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1441.881373] Lustre: Skipped 1 previous similar message [ 1442.968752] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1442.971989] Lustre: Skipped 2 previous similar messages [ 1445.096338] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1445.099222] Lustre: Skipped 5 previous similar messages [ 1449.262831] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1449.266980] Lustre: Skipped 10 previous similar messages [ 1457.578283] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1457.581314] Lustre: Skipped 20 previous similar messages [ 1473.887791] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1473.890364] Lustre: Skipped 39 previous similar messages [ 1484.464631] Lustre: 84946:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1485.668762] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:17:48 (1781029068) [ 1486.400475] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1487.200223] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:17:50 (1781029070) [ 1489.890287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1489.895049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.900907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1492.962181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1492.962525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1492.962888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.975446] Lustre: Skipped 1 previous similar message [ 1494.815766] Lustre: server umount lustre-OST0000 complete [ 1498.081775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.082941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1498.087157] Lustre: Skipped 1 previous similar message [ 1502.370851] Lustre: server umount lustre-MDT0000 complete [ 1503.200974] LustreError: 77848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1503.207919] LustreError: 77848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1504.064611] LustreError: 77053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029089 with bad export cookie 12714652625836234941 [ 1504.069983] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1504.074234] LustreError: 77053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1504.235493] Lustre: server umount lustre-MDT0001 complete [ 1508.548238] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1509.687026] Key type lgssc unregistered [ 1509.835318] LNet: 86173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.838636] LNetError: 86173:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.848374] LNet: Removed LNI 192.168.203.150@tcp [ 1510.203997] Key type .llcrypt unregistered [ 1510.205554] Key type ._llcrypt unregistered [ 1511.701689] Key type ._llcrypt registered [ 1511.703725] Key type .llcrypt registered [ 1511.747192] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1513.243937] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules [ 1513.633582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1513.689591] alg: No test for adler32 (adler32-zlib) [ 1514.555730] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 1514.663625] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1516.255244] Key type lgssc registered [ 1516.831705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1517.333915] LustreError: 87292:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1525.718182] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1529.862909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1538.470035] LustreError: 87292:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1538.479377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.679736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.074610] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1544.586549] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:18:47 (1781029127) [ 1546.359205] LustreError: 88330:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1546.363457] LustreError: 88330:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1546.366935] LustreError: 88330:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1546.502622] Lustre: server umount lustre-MDT0000 complete [ 1551.589713] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1552.760446] Key type lgssc unregistered [ 1552.894607] LNet: 89835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.897410] LNetError: 89835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.908274] LNet: Removed LNI 192.168.203.150@tcp [ 1553.216782] Key type .llcrypt unregistered [ 1553.217931] Key type ._llcrypt unregistered [ 1561.130931] Key type ._llcrypt registered [ 1561.132156] Key type .llcrypt registered [ 1561.177715] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1561.530578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1561.569884] alg: No test for adler32 (adler32-zlib) [ 1562.431448] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 1562.521837] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1564.103122] Key type lgssc registered [ 1564.512605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1567.361772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1567.370702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.553904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.946713] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1573.463763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1573.714179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1575.156419] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1577.800454] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1579.299558] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1582.080069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1582.327139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.463194] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1587.055322] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1620.331379] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:20:03 (1781029203) [ 1620.999899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1621.672097] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:20:05 (1781029205) [ 1624.546819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1624.547874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1624.552749] Lustre: Skipped 2 previous similar messages [ 1624.555883] Lustre: Skipped 1 previous similar message [ 1628.890383] Lustre: server umount lustre-MDT0000 complete [ 1629.664321] LustreError: 91013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.673087] LustreError: 91013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1630.212347] LustreError: 91784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029215 with bad export cookie 16172692141220466172 [ 1630.214605] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1630.218025] LustreError: 91784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1630.342726] Lustre: server umount lustre-MDT0001 complete [ 1641.984885] Lustre: server umount lustre-OST0000 complete [ 1646.450872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.659267] LustreError: 95111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.745801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.471500] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1652.194354] LustreError: 95112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.636727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.964416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1654.595864] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1657.168056] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1658.584638] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1663.615172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1663.709323] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1663.713479] LustreError: 95101:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1663.727907] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1663.732907] LustreError: 97063:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1663.736693] LustreError: 97063:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1663.739681] LustreError: 97063:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1663.771994] Lustre: server umount lustre-OST0000 complete [ 1663.774233] LustreError: 97063:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1667.041951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1667.118536] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1667.123397] LustreError: 95100:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1667.128378] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1667.133230] LustreError: 97525:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1667.136675] LustreError: 97525:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1667.139281] LustreError: 97525:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1667.165213] Lustre: server umount lustre-OST0000 complete [ 1667.167236] LustreError: 97525:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1670.786350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1670.872535] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1670.874938] Lustre: Found index 0 for lustre-OST0000, updating log [ 1670.877930] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1670.905768] Lustre: lustre-OST0000: new disk, initializing [ 1670.907853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1670.910648] Lustre: 98053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1671.022419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.260314] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1675.194285] Lustre: Failing over lustre-OST0000 [ 1675.256763] Lustre: server umount lustre-OST0000 complete [ 1677.729268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.729884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1677.734318] Lustre: Skipped 1 previous similar message [ 1682.912722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1682.915263] Lustre: Skipped 2 previous similar messages [ 1688.031943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1688.035060] Lustre: Skipped 1 previous similar message [ 1691.103179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1691.162815] Lustre: server umount lustre-MDT0000 complete [ 1692.498388] LustreError: 97083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029277 with bad export cookie 16172692141220468902 [ 1692.503277] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.503721] LustreError: 97083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1692.626236] Lustre: server umount lustre-MDT0001 complete [ 1695.555992] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:21:18 (1781029278) [ 1700.837690] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1704.916290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.097783] LustreError: 100456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1705.135312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.512697] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1709.914441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1711.630506] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1713.045589] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1715.435528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1715.686313] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1717.753712] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1720.804579] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1720.808188] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 1720.813739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1721.106850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1721.140137] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1721.171481] Lustre: lustre-OST0001: new disk, initializing [ 1721.172902] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1721.175718] Lustre: 102862:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1721.584649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1721.589639] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1721.627038] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1723.322162] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1724.564290] Lustre: lustre-OST0000: Denying connection for new client efe31daf-b803-47db-bc06-873b865081fb (at 192.168.203.50@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 1724.570419] Lustre: Skipped 1 previous similar message [ 1726.951106] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 1726.956640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1726.958809] LustreError: Skipped 1 previous similar message [ 1727.045300] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1730.068151] Lustre: lustre-OST0000: Denying connection for new client efe31daf-b803-47db-bc06-873b865081fb (at 192.168.203.50@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:50 [ 1731.048858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1731.057426] Lustre: lustre-OST0000: Recovery over after 0:11, of 1 clients 1 recovered and 0 were evicted. [ 1731.057587] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1731.059908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1732.855787] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1735.197798] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [ 1736.165650] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.166512] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1736.169858] Lustre: Skipped 1 previous similar message [ 1740.439919] Lustre: server umount lustre-MDT0001 complete [ 1741.280574] LustreError: 100456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.286545] LustreError: 100456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1743.665401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.702976] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1743.704774] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1743.722025] Lustre: 100446:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1743.968671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1743.970724] Lustre: Skipped 3 previous similar messages [ 1745.463145] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1745.637510] LustreError: 104659:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1745.641078] LustreError: 104659:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1745.644632] LustreError: 100448:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1745.671637] LustreError: 104665:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.150@tcp: cfg command failed: rc = -17 [ 1745.675033] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1745.675033] [ 1749.118819] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 14:22:12 (1781029332) [ 1749.478876] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1755.262243] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1761.860943] Lustre: 106519:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1770.699204] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1771.979849] Lustre: 107651:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1820.640552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1820.644264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.649322] Lustre: Skipped 1 previous similar message [ 1820.651513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1820.654379] Lustre: Skipped 4 previous similar messages [ 1825.245755] Lustre: server umount lustre-MDT0000 complete [ 1826.271545] LustreError: 100451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.276301] LustreError: 100451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1826.447777] LustreError: 102861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029411 with bad export cookie 16172692141220469749 [ 1826.450715] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1826.451635] LustreError: 102861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1826.590249] Lustre: server umount lustre-MDT0001 complete [ 1837.587514] Lustre: server umount lustre-OST0000 complete [ 1848.161683] Lustre: server umount lustre-OST0001 complete [ 1850.192570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1853.538908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1856.480485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1859.653474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1866.020825] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1869.521315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.530057] Lustre: lustre-MDT0000: reset Object Index mappings [ 1869.687123] LustreError: 111561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.713843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.091544] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1874.417036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.425150] Lustre: lustre-MDT0001: reset Object Index mappings [ 1875.908416] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1877.288714] Lustre: 112773:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1879.525677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1881.714785] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1884.908720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 1884.910070] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 1884.923679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1887.163889] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1890.280948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 1890.282972] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1890.767514] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1892.101888] Lustre: 114708:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1908.424641] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:24:51 (1781029491) [ 1941.474926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1941.475052] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.477260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1941.477268] Lustre: Skipped 4 previous similar messages [ 1941.489209] Lustre: Skipped 6 previous similar messages [ 1944.165142] Lustre: server umount lustre-MDT0000 complete [ 1945.506444] LustreError: 111543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029530 with bad export cookie 16172692141220516719 [ 1945.513455] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1946.593099] LustreError: 113971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.599289] LustreError: 113971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1951.763928] Lustre: server umount lustre-MDT0001 complete [ 1959.500279] Lustre: server umount lustre-OST0000 complete [ 1975.263122] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1975.342099] Lustre: server umount lustre-OST0001 complete [ 1977.629970] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 1981.132533] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1993.989686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1994.088823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1994.105630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1994.144121] Lustre: lustre-MDT0000: new disk, initializing [ 1994.189643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1995.745737] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1999.401043] Lustre: server umount lustre-MDT0000 complete [ 2002.949601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2003.101347] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2003.109639] Lustre: 118974:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2003.115812] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2003.200530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.202919] Lustre: Skipped 4 previous similar messages [ 2004.855675] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2007.405510] Lustre: server umount lustre-MDT0000 complete [ 2011.117519] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:26:34 (1781029594) [ 2011.796099] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2012.544330] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:26:35 (1781029595) [ 2019.665878] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2023.660573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.382185] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2029.299098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.350978] Lustre: 121035:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2029.380810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2029.383197] Lustre: Skipped 1 previous similar message [ 2029.422569] Lustre: lustre-MDT0001: new disk, initializing [ 2029.476403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2029.480894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2031.010173] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2033.981359] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2036.612421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2036.730904] Lustre: lustre-OST0000: new disk, initializing [ 2036.733907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2036.739556] Lustre: 122674:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2037.006196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2037.010586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2037.057020] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2038.998628] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2044.149501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2044.224563] Lustre: 123579:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2045.840555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2045.844932] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2045.877907] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2046.519394] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2051.971287] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2053.402240] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2225.611959] Lustre: 129990:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2225.615599] Lustre: 129990:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2225.617783] Lustre: 129990:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2225.619970] Lustre: 129990:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2225.621826] Lustre: 129990:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2225.623661] Lustre: 129990:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2305.308344] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:31:28 (1781029888) [ 2318.958517] Lustre: server umount lustre-MDT0000 complete [ 2320.172089] LustreError: 123578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029905 with bad export cookie 16172692141220571235 [ 2320.175971] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2320.176766] LustreError: 123578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2320.352529] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.353886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2320.354200] LustreError: 124412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.354210] LustreError: 124412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2320.355604] Lustre: Skipped 5 previous similar messages [ 2320.367517] Lustre: Skipped 8 previous similar messages [ 2326.403623] Lustre: server umount lustre-MDT0001 complete [ 2333.756347] Lustre: server umount lustre-OST0000 complete [ 2341.135731] Lustre: server umount lustre-OST0001 complete [ 2343.282584] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2346.431274] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2358.764757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.878500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2358.891803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2358.893956] Lustre: Skipped 1 previous similar message [ 2358.935351] Lustre: lustre-MDT0000: new disk, initializing [ 2358.937434] Lustre: Skipped 1 previous similar message [ 2358.968288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.970651] Lustre: Skipped 4 previous similar messages [ 2358.976562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2360.411080] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2365.110470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.157462] Lustre: 132967:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2365.261983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2366.623456] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2370.343667] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2371.636503] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2374.011074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2374.143952] Lustre: 134803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2376.128801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2376.132631] Lustre: Skipped 1 previous similar message [ 2376.135642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2376.183293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2376.208385] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2379.833208] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2383.304820] Lustre: lustre-MDD0000: changelog on [ 2384.336363] Lustre: lustre-MDD0001: changelog on [ 2706.019893] Lustre: 138039:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2735.044034] Lustre: lustre-MDD0001: changelog off [ 2736.237827] Lustre: lustre-MDD0000: changelog off [ 2739.681353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.682024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2739.683434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2739.683440] Lustre: Skipped 2 previous similar messages [ 2739.685038] Lustre: Skipped 1 previous similar message [ 2739.687237] LustreError: Skipped 1 previous similar message [ 2743.959306] Lustre: server umount lustre-OST0000 complete [ 2749.407871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2751.424126] Lustre: server umount lustre-MDT0000 complete [ 2752.630098] LustreError: 132960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030337 with bad export cookie 16172692141221339548 [ 2752.633275] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2752.633601] LustreError: 132960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2752.754045] Lustre: server umount lustre-MDT0001 complete [ 2756.263491] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2757.294947] Key type lgssc unregistered [ 2757.428324] LNet: 143418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2757.431937] LNetError: 143418:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2757.440313] LNet: Removed LNI 192.168.203.150@tcp [ 2757.746097] Key type .llcrypt unregistered [ 2757.747164] Key type ._llcrypt unregistered [ 2764.029267] Key type ._llcrypt registered [ 2764.030920] Key type .llcrypt registered [ 2764.071579] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2770.444533] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2770.792974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2770.854604] alg: No test for adler32 (adler32-zlib) [ 2771.713153] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 2771.796085] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2773.375125] Key type lgssc registered [ 2773.732790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2785.519458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2785.525736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.629435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2786.641586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2786.684156] Lustre: lustre-MDT0000: new disk, initializing [ 2786.713156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.720204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2788.101989] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2792.799993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.843230] Lustre: 146556:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2792.866477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2792.868319] Lustre: Skipped 1 previous similar message [ 2792.902983] Lustre: lustre-MDT0001: new disk, initializing [ 2792.934060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2792.945819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2792.949406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2794.268600] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2797.846940] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2799.118758] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2801.437273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2801.543627] Lustre: lustre-OST0000: new disk, initializing [ 2801.545379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2801.548103] Lustre: 148392:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2801.576223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.980853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2801.983816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2802.013081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2803.447870] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2806.858262] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2809.074452] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_module obdecho/obdecho [ 2809.685021] LustreError: 149314:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2809.687734] LustreError: 149314:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2810.270606] LustreError: 149410:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 2813.407982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2813.410546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.415101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2817.506684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2817.508209] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.512560] Lustre: Skipped 1 previous similar message [ 2819.487367] Lustre: server umount lustre-MDT0000 complete [ 2820.724258] LustreError: 146548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030405 with bad export cookie 14056272395675546069 [ 2820.727854] LustreError: 146548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2820.728456] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2820.855614] Lustre: server umount lustre-MDT0001 complete [ 2832.408604] Lustre: server umount lustre-OST0000 complete [ 2839.261258] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2842.799490] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2854.683890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.784233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2854.795729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2854.838143] Lustre: lustre-MDT0000: new disk, initializing [ 2854.864170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.870508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2856.235920] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2860.606484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.645884] Lustre: 153085:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2860.672700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2860.674212] Lustre: Skipped 1 previous similar message [ 2860.705995] Lustre: lustre-MDT0001: new disk, initializing [ 2860.741934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2860.755719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2860.760589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2862.141537] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2865.552955] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2866.722492] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2868.922549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2869.021560] Lustre: lustre-OST0000: new disk, initializing [ 2869.023206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2869.026269] Lustre: 154925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2869.051479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2869.482153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2869.486620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2869.517717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2870.943125] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2874.378649] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2875.734430] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2875.813716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2877.093492] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2877.168437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2879.969731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2879.970824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.972142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2879.972147] Lustre: Skipped 2 previous similar messages [ 2879.972577] LustreError: Skipped 1 previous similar message [ 2879.976260] Lustre: Skipped 1 previous similar message [ 2883.921687] Lustre: server umount lustre-OST0000 complete [ 2886.623851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2886.626234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.629898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2886.631503] Lustre: Skipped 1 previous similar message [ 2891.290525] Lustre: server umount lustre-MDT0000 complete [ 2892.480907] LustreError: 153077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030477 with bad export cookie 14056272395675547014 [ 2892.483758] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2892.484190] LustreError: 153077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2892.597372] Lustre: server umount lustre-MDT0001 complete [ 2893.755414] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:41:17 (1781030477) [ 2894.317863] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2894.908276] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:41:18 (1781030478) [ 2897.490052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2897.682666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.010439] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2902.334056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2903.755790] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2905.897260] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2907.111151] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2909.441662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2911.400642] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2913.492393] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2928.097659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2928.098610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.104161] Lustre: Skipped 1 previous similar message [ 2928.106822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.109243] Lustre: Skipped 4 previous similar messages [ 2932.068577] Lustre: server umount lustre-MDT0000 complete [ 2933.217758] LustreError: 157231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.225383] LustreError: 157231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2933.363981] LustreError: 158940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030518 with bad export cookie 14056272395675547882 [ 2933.368019] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2933.368717] LustreError: 158940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2934.290747] LustreError: 158620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.336138] LustreError: 157228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.336351] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.347205] Lustre: Skipped 2 previous similar messages [ 2938.349604] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2938.352136] Lustre: Skipped 2 previous similar messages [ 2939.577610] Lustre: server umount lustre-MDT0001 complete [ 2941.738846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2945.886979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.962759] LustreError: 160894:0:(llog.c:1646:llog_backup()) MGC192.168.203.150@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2945.965635] Lustre: 160894:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.150@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2958.816439] LustreError: 160899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.822035] LustreError: 160899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2963.935586] LustreError: 160901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.175895] LustreError: 160899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.183041] LustreError: 160899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2990.610421] LustreError: 160900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.614443] LustreError: 160900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 3006.559267] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3006.561118] Lustre: 160894:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.150@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3024.351913] LustreError: 160901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.356049] LustreError: 160901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 3047.399631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3047.403201] Lustre: Skipped 2 previous similar messages [ 3048.515916] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3051.521329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.515147] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3053.535678] LustreError: 144783:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ce3b7b5c x1867545686671488/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3054.945681] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3057.163677] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3057.846073] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3058.661385] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3059.140878] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3061.756672] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3068.561212] Lustre: server umount lustre-MDT0000 complete [ 3068.896921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3068.898029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3068.902869] Lustre: Skipped 1 previous similar message [ 3069.729209] LustreError: 158940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030654 with bad export cookie 14056272395675554322 [ 3069.731905] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3069.732726] LustreError: 158940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3069.835690] Lustre: server umount lustre-MDT0001 complete [ 3081.307642] Lustre: server umount lustre-OST0000 complete [ 3088.283565] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3091.836511] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3104.297812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.396744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3104.409282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3104.446459] Lustre: lustre-MDT0000: new disk, initializing [ 3104.478259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3105.835687] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3110.318872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.357936] Lustre: 165801:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3110.452660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3111.643802] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3114.918338] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3116.086553] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3118.242796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3118.343430] Lustre: 167636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3118.368233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.371169] Lustre: Skipped 3 previous similar messages [ 3119.026974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3119.057785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3120.072051] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3123.381613] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3124.713038] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3124.780119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3126.004037] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3126.074709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3129.312758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3129.313130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.314913] LustreError: Skipped 1 previous similar message [ 3129.318140] Lustre: Skipped 1 previous similar message [ 3129.318748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3129.322421] Lustre: Skipped 2 previous similar messages [ 3132.752957] Lustre: server umount lustre-OST0000 complete [ 3148.255111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3148.322623] Lustre: server umount lustre-MDT0000 complete [ 3149.562168] LustreError: 166626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030734 with bad export cookie 14056272395675571122 [ 3149.565509] LustreError: 166626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3149.566365] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3149.677258] Lustre: server umount lustre-MDT0001 complete [ 3152.645919] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:45:36 (1781030736) [ 3154.845110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.020150] LustreError: 169949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.025416] LustreError: 169949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 3156.287584] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3159.520721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.085232] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3163.006670] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3164.127908] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3166.437496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3168.580403] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3170.823064] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3185.664334] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:46:09 (1781030769) [ 3190.247035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.248678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.249453] LustreError: Skipped 1 previous similar message [ 3190.251689] Lustre: Skipped 7 previous similar messages [ 3192.782543] Lustre: server umount lustre-OST0000 complete [ 3195.360889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.367021] Lustre: Skipped 6 previous similar messages [ 3200.216725] Lustre: server umount lustre-MDT0000 complete [ 3201.466463] LustreError: 169974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030786 with bad export cookie 14056272395675571990 [ 3201.471686] LustreError: 169974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3201.472292] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3201.608312] Lustre: server umount lustre-MDT0001 complete [ 3205.372742] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3206.409213] Key type lgssc unregistered [ 3206.550418] LNet: 173484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3206.554543] LNetError: 173484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3206.566440] LNet: Removed LNI 192.168.203.150@tcp [ 3206.894392] Key type .llcrypt unregistered [ 3206.895627] Key type ._llcrypt unregistered [ 3214.799318] Key type ._llcrypt registered [ 3214.800350] Key type .llcrypt registered [ 3214.836849] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3215.134484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3215.142337] alg: No test for adler32 (adler32-zlib) [ 3215.993394] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 3216.071925] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3217.655111] Key type lgssc registered [ 3218.034375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3220.785815] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3220.793554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.255243] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3241.375205] LustreError: 174686:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 3246.500128] LustreError: 174686:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3246.505228] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3246.507850] LustreError: MGC192.168.203.150@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3246.511744] LustreError: 174686:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3246.514975] LustreError: 174686:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 3246.518052] LustreError: 174686:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 3246.545146] Lustre: server umount lustre-OST0000 complete [ 3246.546546] LustreError: 174686:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3249.211236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.355619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.531637] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3254.385807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.492457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3255.669343] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3257.640621] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3258.719717] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3260.796219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3260.870769] LustreError: 176866:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3260.874041] Lustre: 176866:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.150@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3260.918514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.672806] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3264.589022] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3270.114349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.117217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3270.117615] Lustre: Skipped 2 previous similar messages [ 3270.119289] Lustre: Skipped 1 previous similar message [ 3270.121345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3271.378683] Lustre: server umount lustre-MDT0000 complete [ 3272.516169] LustreError: 175165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030857 with bad export cookie 7449045410404286275 [ 3272.518382] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3272.519156] LustreError: 175165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3272.636215] Lustre: server umount lustre-MDT0001 complete [ 3283.098641] Lustre: server umount lustre-OST0000 complete [ 3285.957691] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3288.861843] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3300.146664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.247266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3300.259587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3300.293133] Lustre: lustre-MDT0000: new disk, initializing [ 3300.316648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.323732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3301.617727] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3305.917507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.956593] Lustre: 180285:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3305.978904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3305.981247] Lustre: Skipped 1 previous similar message [ 3306.022516] Lustre: lustre-MDT0001: new disk, initializing [ 3306.053214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3306.064014] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3306.067080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3307.251143] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3310.618938] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3311.858064] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3314.133309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3314.242027] Lustre: lustre-OST0000: new disk, initializing [ 3314.243656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3314.246424] Lustre: 182124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3314.276938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.240329] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3317.943133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3317.946390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3317.978371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3319.799164] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3321.365438] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3321.452103] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3322.906968] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3322.988936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3325.714585] Lustre: server umount lustre-OST0000 complete [ 3328.481269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.481849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3328.487687] Lustre: Skipped 1 previous similar message [ 3333.600209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3333.602787] Lustre: Skipped 2 previous similar messages [ 3337.726110] Lustre: server umount lustre-MDT0000 complete [ 3338.719902] LustreError: 180297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.724438] LustreError: 180297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3338.901179] LustreError: 180277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030923 with bad export cookie 7449045410404286968 [ 3338.904032] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3338.905452] LustreError: 180277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3348.495432] Lustre: server umount lustre-MDT0001 complete [ 3351.234915] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:48:54 (1781030934) [ 3357.083318] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3358.101518] Key type lgssc unregistered [ 3358.230342] LNet: 184949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3358.232608] LNetError: 184949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3358.238513] LNet: Removed LNI 192.168.203.150@tcp [ 3358.541926] Key type .llcrypt unregistered [ 3358.543085] Key type ._llcrypt unregistered [ 3365.276452] Key type ._llcrypt registered [ 3365.277551] Key type .llcrypt registered [ 3365.314550] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3365.733299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3365.768820] alg: No test for adler32 (adler32-zlib) [ 3366.628246] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 3366.723211] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3368.311141] Key type lgssc registered [ 3368.682251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3371.424075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3371.431293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.598982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3373.889709] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3377.011045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.126668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3378.384020] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3380.381139] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3381.511184] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3383.970211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.054773] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3384.056724] LustreError: 187861:0:(llog.c:1687:llog_backup()) MGC192.168.203.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3384.059735] Lustre: 187861:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.150@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3384.098928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.962895] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3388.169925] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3392.482399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3392.482812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3392.483643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3392.487554] Lustre: Skipped 2 previous similar messages [ 3397.601418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.603197] Lustre: Skipped 4 previous similar messages [ 3402.721113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3402.723010] Lustre: Skipped 2 previous similar messages [ 3403.231153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3403.295896] Lustre: server umount lustre-MDT0000 complete [ 3404.559126] LustreError: 186107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030989 with bad export cookie 12068534343422975076 [ 3404.563425] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3404.564109] LustreError: 186107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3404.672435] Lustre: server umount lustre-MDT0001 complete [ 3415.191103] Lustre: server umount lustre-OST0000 complete [ 3418.244038] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3421.507108] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3433.316253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.421089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3433.434984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3433.483347] Lustre: lustre-MDT0000: new disk, initializing [ 3433.510587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.518882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3434.864914] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3439.404056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.454380] Lustre: 191280:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3439.471988] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3439.473214] Lustre: Skipped 1 previous similar message [ 3439.474531] LustreError: 192092:0:(llog.c:1687:llog_backup()) MGC192.168.203.150@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3439.477073] LustreError: 192092:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3439.478962] Lustre: 192092:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.150@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3439.486328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3439.488170] Lustre: Skipped 1 previous similar message [ 3439.523745] Lustre: lustre-MDT0001: new disk, initializing [ 3439.557056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3439.570104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3439.574522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3440.932652] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3444.452649] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3445.668573] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3448.018627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3448.136676] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3448.137886] Lustre: Skipped 1 previous similar message [ 3448.139409] LustreError: 193093:0:(llog.c:1687:llog_backup()) MGC192.168.203.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3448.143789] LustreError: 193093:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3448.147028] Lustre: 193093:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.150@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3448.151593] Lustre: 193093:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3448.162098] Lustre: lustre-OST0000: new disk, initializing [ 3448.163648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3448.165865] Lustre: 193117:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3448.196453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3450.224169] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3453.778341] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3455.233066] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3457.527842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3457.532689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3457.558790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3458.362577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3459.707401] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3459.789359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3462.626926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.626931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3462.628355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3462.630927] Lustre: Skipped 1 previous similar message [ 3466.582884] Lustre: server umount lustre-OST0000 complete [ 3470.303895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3470.307338] LustreError: Skipped 1 previous similar message [ 3470.309295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3470.315166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3470.317579] Lustre: Skipped 1 previous similar message [ 3472.864782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.071551] Lustre: server umount lustre-MDT0000 complete [ 3475.343228] LustreError: 191316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031060 with bad export cookie 12068534343422975776 [ 3475.345148] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.348383] LustreError: 191316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3475.471456] Lustre: server umount lustre-MDT0001 complete [ 3478.505601] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:51:01 (1781031061) [ 3484.209714] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3487.871562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3488.013054] LustreError: 196144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3488.021027] LustreError: 196144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3488.028121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.336195] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3492.549428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3494.002141] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3495.383413] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3497.688865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3499.585532] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3502.615649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3502.688721] Lustre: lustre-OST0001: new disk, initializing [ 3502.690707] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3502.693622] Lustre: 198537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3503.109978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3503.114501] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3503.151662] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3504.632661] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3510.275531] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3515.894563] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3516.459400] Lustre: DEBUG MARKER: ADD OST3 [ 3517.993169] loop: module loaded [ 3518.009637] loop0: detected capacity change from 0 to 204800000 [ 3518.121763] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3518.126891] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3518.131336] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3519.999269] loop0: detected capacity change from 0 to 204800000 [ 3521.790963] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3521.851383] Lustre: lustre-OST0003: new disk, initializing [ 3521.852714] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3521.854691] Lustre: 200302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3521.889606] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3521.892441] Lustre: Skipped 3 previous similar messages [ 3522.946983] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3522.984975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3522.988516] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3522.999148] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3523.018720] Lustre: DEBUG MARKER: STOP OST3 [ 3523.655434] Lustre: Failing over lustre-OST0003 [ 3523.678128] Lustre: lustre-OST0003: Not available for connect from 192.168.203.50@tcp (stopping) [ 3523.680961] Lustre: Skipped 2 previous similar messages [ 3523.699996] Lustre: server umount lustre-OST0003 complete [ 3523.766292] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3524.959116] LustreError: 200504:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3527.865435] Lustre: DEBUG MARKER: START OST3 again [ 3528.161445] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3528.164431] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3528.167214] LustreError: 197689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3528.168902] Lustre: Skipped 1 previous similar message [ 3529.992163] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3530.075582] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3531.233985] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3531.241630] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3531.241827] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3531.893702] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3545.447068] Lustre: Failing over lustre-OST0003 [ 3545.489326] Lustre: server umount lustre-OST0003 complete [ 3545.574407] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.579829] LustreError: 198547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3545.586766] LustreError: 198547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3550.692633] LustreError: 197691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3550.696888] LustreError: 197691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3555.813213] LustreError: 202443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3555.817199] LustreError: 202443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3559.392264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3559.395085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3563.994947] Lustre: server umount lustre-MDT0000 complete [ 3565.104982] LustreError: 196124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031150 with bad export cookie 12068534343422976637 [ 3565.107784] LustreError: 196124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3565.108632] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3566.047939] LustreError: 196139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.048453] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.052283] LustreError: 196139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3566.055208] Lustre: Skipped 6 previous similar messages [ 3571.313536] Lustre: server umount lustre-MDT0001 complete [ 3572.606454] Lustre: server umount lustre-OST0000 complete [ 3573.903267] Lustre: server umount lustre-OST0001 complete [ 3575.960824] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3578.635087] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3590.925415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.021810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3591.033992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3591.072074] Lustre: lustre-MDT0000: new disk, initializing [ 3591.098202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.101279] Lustre: Skipped 1 previous similar message [ 3591.107292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3592.429541] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3596.949635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.995506] Lustre: 205741:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3597.095183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3598.434429] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3601.812656] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3603.024092] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3605.239625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3605.356276] Lustre: 207578:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3607.031384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3607.068692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3607.193480] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3610.586925] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3612.051118] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3612.138094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3613.487963] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3613.558611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3617.250406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3617.250616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3617.251143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3617.251148] Lustre: Skipped 8 previous similar messages [ 3617.253823] Lustre: Skipped 1 previous similar message [ 3617.255825] LustreError: Skipped 1 previous similar message [ 3620.299276] Lustre: server umount lustre-OST0000 complete [ 3627.672570] Lustre: server umount lustre-MDT0000 complete [ 3628.853203] LustreError: 205734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031213 with bad export cookie 12068534343422978891 [ 3628.855939] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3628.856337] LustreError: 205734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3628.969609] Lustre: server umount lustre-MDT0001 complete [ 3630.103742] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:53:33 (1781031213) [ 3635.021633] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3638.053771] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3649.752361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.835357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3649.845120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3649.847551] Lustre: Skipped 3 previous similar messages [ 3649.887501] Lustre: lustre-MDT0000: new disk, initializing [ 3649.890368] Lustre: Skipped 2 previous similar messages [ 3649.921732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3649.925850] Lustre: Skipped 2 previous similar messages [ 3651.228661] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3655.714802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.754824] Lustre: 212047:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3655.856606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3657.096147] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3660.451248] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3661.590378] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3663.737178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3663.842545] Lustre: 213885:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3665.699762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3665.734395] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3669.087849] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3670.458914] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3670.536871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3671.845131] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3671.907126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3674.578410] Lustre: server umount lustre-OST0000 complete [ 3685.375682] Lustre: server umount lustre-MDT0000 complete [ 3686.368182] LustreError: 212058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.374318] LustreError: 212058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3686.715836] LustreError: 212038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031271 with bad export cookie 12068534343422979773 [ 3686.720585] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3686.720745] LustreError: 212038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3696.144546] Lustre: server umount lustre-MDT0001 complete [ 3698.960193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3700.423496] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3703.597177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.988884] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3707.149734] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3708.334447] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3710.646093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3712.661888] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3714.727055] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3734.497783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3734.498996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3734.500219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3734.500223] Lustre: Skipped 9 previous similar messages [ 3734.502199] LustreError: Skipped 1 previous similar message [ 3734.506150] Lustre: Skipped 5 previous similar messages [ 3737.551579] Lustre: server umount lustre-OST0000 complete [ 3744.922862] Lustre: server umount lustre-MDT0000 complete [ 3746.173547] LustreError: 215986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031331 with bad export cookie 12068534343422980655 [ 3746.177378] LustreError: 215986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3746.177701] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3746.303205] Lustre: server umount lustre-MDT0001 complete [ 3850.334963] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3851.262192] Key type lgssc unregistered [ 3851.386388] LNet: 219479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3851.389136] LNetError: 219479:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3851.397248] LNet: Removed LNI 192.168.203.150@tcp [ 3851.695111] Key type .llcrypt unregistered [ 3851.696158] Key type ._llcrypt unregistered [ 3859.481288] Key type ._llcrypt registered [ 3859.482308] Key type .llcrypt registered [ 3859.518894] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3865.613672] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3865.940521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3865.993555] alg: No test for adler32 (adler32-zlib) [ 3866.846718] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 3866.925219] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3868.503137] Key type lgssc registered [ 3868.857265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3880.275569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3880.280236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.375724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3881.388381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3881.429810] Lustre: lustre-MDT0000: new disk, initializing [ 3881.454599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3881.461576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3882.807273] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3887.311470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.355887] Lustre: 222769:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3887.381101] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3887.382904] Lustre: Skipped 1 previous similar message [ 3887.425953] Lustre: lustre-MDT0001: new disk, initializing [ 3887.456678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3887.467846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3887.470988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3888.762612] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3892.283105] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3893.536379] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3895.868173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3895.982856] Lustre: lustre-OST0000: new disk, initializing [ 3895.984880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3895.987253] Lustre: 224608:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3896.021930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.873185] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3898.614796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3898.618064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3898.646029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3901.388778] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3902.782413] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3902.848782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3904.137113] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3904.200610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3909.090157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3909.090172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3909.091723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3909.092885] LustreError: Skipped 1 previous similar message [ 3909.098388] Lustre: Skipped 1 previous similar message [ 3914.210107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3914.211993] Lustre: Skipped 2 previous similar messages [ 3919.327087] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3919.329807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3919.334429] Lustre: Skipped 1 previous similar message [ 3919.376379] Lustre: server umount lustre-OST0000 complete [ 3923.423765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3923.426616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.430827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3924.449483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.745032] Lustre: server umount lustre-MDT0000 complete [ 3927.985025] LustreError: 222763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031512 with bad export cookie 14414241542485427558 [ 3927.988253] LustreError: 222763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3927.988860] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3928.109624] Lustre: server umount lustre-MDT0001 complete [ 3934.116749] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3937.405215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.551777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3938.850504] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3941.967070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.086973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3943.421841] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3944.790235] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3946.997740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3949.077710] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3952.417826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3952.487440] Lustre: lustre-OST0001: new disk, initializing [ 3952.489717] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3952.491951] Lustre: 229827:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3952.521350] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3952.523552] Lustre: Skipped 1 previous similar message [ 3954.530835] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3955.077705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3955.080891] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3955.120533] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3960.430532] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3967.096585] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3970.853499] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:59:14 (1781031554) [ 3973.089286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3973.092034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.097407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3973.099144] Lustre: Skipped 2 previous similar messages [ 3977.687133] Lustre: server umount lustre-MDT0000 complete [ 3978.899678] LustreError: 227405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031563 with bad export cookie 14414241542485428426 [ 3978.905049] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3978.905249] LustreError: 227405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3979.031104] Lustre: server umount lustre-MDT0001 complete [ 3990.977928] Lustre: server umount lustre-OST0000 complete [ 4002.529420] Lustre: server umount lustre-OST0001 complete [ 4004.648856] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4007.741740] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4019.854191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.957744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4019.968853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4020.009844] Lustre: lustre-MDT0000: new disk, initializing [ 4020.034427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4020.042784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4021.357043] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4025.794307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4025.833273] Lustre: 233761:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4025.939825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4027.179416] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4030.618673] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4031.823254] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4034.121956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4034.239799] Lustre: lustre-OST0000: new disk, initializing [ 4034.241253] Lustre: Skipped 1 previous similar message [ 4034.242732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4034.244733] Lustre: Skipped 2 previous similar messages [ 4034.246755] Lustre: 235598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4036.098895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4036.101508] Lustre: Skipped 1 previous similar message [ 4036.103525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4036.152725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4036.236423] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4039.725615] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4041.133972] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4041.212650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4042.565374] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4042.650149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4046.306575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4046.307510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.308794] LustreError: Skipped 1 previous similar message [ 4046.313825] Lustre: Skipped 3 previous similar messages [ 4046.316513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4046.319184] Lustre: Skipped 5 previous similar messages [ 4049.437727] Lustre: server umount lustre-OST0000 complete [ 4051.427472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4065.247097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4065.311820] Lustre: server umount lustre-MDT0000 complete [ 4066.505198] LustreError: 233752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031651 with bad export cookie 14414241542485430519 [ 4066.507556] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4066.508348] LustreError: 233752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4066.619911] Lustre: server umount lustre-MDT0001 complete [ 4069.368290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4069.523900] LustreError: 237713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.529212] LustreError: 237713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4069.547265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4069.551182] Lustre: Skipped 2 previous similar messages [ 4070.791636] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4073.881340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.249619] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4077.356424] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4078.525026] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4080.736690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4082.671922] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4084.744957] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4228.578275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.579295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4228.579667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4228.579671] Lustre: Skipped 6 previous similar messages [ 4228.582829] Lustre: Skipped 3 previous similar messages [ 4234.511226] Lustre: server umount lustre-OST0000 complete [ 4250.079399] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4250.147086] Lustre: server umount lustre-MDT0000 complete [ 4251.398304] LustreError: 239437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031836 with bad export cookie 14414241542485431401 [ 4251.402254] LustreError: 239437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4251.403005] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4251.518643] Lustre: server umount lustre-MDT0001 complete [ 4349.453214] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4350.446208] Key type lgssc unregistered [ 4350.569259] LNet: 241117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4350.571558] LNetError: 241117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4350.580296] LNet: Removed LNI 192.168.203.150@tcp [ 4350.869128] Key type .llcrypt unregistered [ 4350.870814] Key type ._llcrypt unregistered [ 4358.608371] Key type ._llcrypt registered [ 4358.609406] Key type .llcrypt registered [ 4358.645525] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4364.558672] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4364.842538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4364.859596] alg: No test for adler32 (adler32-zlib) [ 4365.718306] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 4365.800192] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4367.383138] Key type lgssc registered [ 4367.739166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4381.810642] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4385.399450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4385.406272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.491333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4386.502054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4386.538208] Lustre: lustre-MDT0000: new disk, initializing [ 4386.561495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.568182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4387.833954] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4392.250407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.293654] Lustre: 245108:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4392.318708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4392.321824] Lustre: Skipped 1 previous similar message [ 4392.362388] Lustre: lustre-MDT0001: new disk, initializing [ 4392.393571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4392.405598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4392.410123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4393.654111] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4396.246661] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4398.443967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4398.555525] Lustre: lustre-OST0000: new disk, initializing [ 4398.558869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4398.563608] Lustre: 246801:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4398.601542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4400.738707] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4402.049130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4402.053654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4402.093802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4405.124385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4405.188943] Lustre: lustre-OST0001: new disk, initializing [ 4405.190631] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4405.192985] Lustre: 247705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4405.231756] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4407.081626] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4407.681078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4407.686072] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4407.711252] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4412.894477] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4418.596174] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4423.138284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.139960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4423.142611] Lustre: Skipped 1 previous similar message [ 4423.143649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4423.144849] Lustre: Skipped 2 previous similar messages [ 4427.617423] Lustre: server umount lustre-MDT0000 complete [ 4428.790250] LustreError: 246800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032013 with bad export cookie 16902883791870696748 [ 4428.793316] LustreError: 246800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4428.793428] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4428.914600] Lustre: server umount lustre-MDT0001 complete [ 4440.580332] Lustre: server umount lustre-OST0000 complete [ 4452.064206] Lustre: server umount lustre-OST0001 complete [ 4456.546716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4459.697360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4462.404156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4465.588682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4471.607970] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4474.920426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4474.928571] Lustre: lustre-MDT0000: reset Object Index mappings [ 4475.064108] LustreError: 252720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.069568] LustreError: 252720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4475.081710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.312504] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4479.607322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.614264] Lustre: lustre-MDT0001: reset Object Index mappings [ 4481.053080] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4482.388886] Lustre: 253923:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4484.653169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4484.796376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4484.798332] Lustre: Skipped 1 previous similar message [ 4485.795638] LustreError: 254276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.798270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4486.676620] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4489.896310] LustreError: 254460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4489.905337] LustreError: 254460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4490.242631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4492.411279] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4495.848388] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4496.361883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4497.658434] Lustre: 255856:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4500.448746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4500.451603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4500.455126] Lustre: Skipped 2 previous similar messages [ 4500.456971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4504.932223] Lustre: server umount lustre-MDT0000 complete [ 4506.081388] LustreError: 252720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4506.086073] LustreError: 252720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4506.088096] LustreError: 252702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032091 with bad export cookie 16902883791870703524 [ 4506.088400] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4506.200105] Lustre: server umount lustre-MDT0001 complete [ 4517.381117] Lustre: server umount lustre-OST0000 complete [ 4528.859213] Lustre: server umount lustre-OST0001 complete [ 4535.066771] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:08:38 (1781032118) [ 4540.316132] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4543.413317] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4555.480812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.586536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4555.598319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4555.638797] Lustre: lustre-MDT0000: new disk, initializing [ 4555.662825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.665085] Lustre: Skipped 1 previous similar message [ 4555.670788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4557.012832] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4561.422351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.469494] Lustre: 260068:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4561.578432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4562.843457] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4566.291515] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4567.463905] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4569.704237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4569.825619] Lustre: lustre-OST0000: new disk, initializing [ 4569.827008] Lustre: Skipped 1 previous similar message [ 4569.828643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4569.830852] Lustre: Skipped 2 previous similar messages [ 4569.833087] Lustre: 261907:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4571.128327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4571.131500] Lustre: Skipped 1 previous similar message [ 4571.133536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4571.169752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4571.847769] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4575.324516] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4576.658822] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4576.730087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4578.006366] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4578.079260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4581.346285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4581.347108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.347805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4581.347809] Lustre: Skipped 3 previous similar messages [ 4581.349986] LustreError: Skipped 1 previous similar message [ 4581.353917] Lustre: Skipped 4 previous similar messages [ 4584.788766] Lustre: server umount lustre-OST0000 complete [ 4586.464658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.465297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4586.469527] Lustre: Skipped 1 previous similar message [ 4586.471617] Lustre: Skipped 1 previous similar message [ 4591.584138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4591.587933] Lustre: Skipped 1 previous similar message [ 4600.287161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4600.355770] Lustre: server umount lustre-MDT0000 complete [ 4601.587343] LustreError: 260061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032186 with bad export cookie 16902883791870705596 [ 4601.590952] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4601.594464] LustreError: 260061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4601.711417] Lustre: server umount lustre-MDT0001 complete [ 4607.196158] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4610.415061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.546782] LustreError: 264719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.561572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4610.563638] Lustre: Skipped 2 previous similar messages [ 4611.822390] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4615.074704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.564430] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4617.949866] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4620.214843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4622.233411] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4625.745372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4625.809424] Lustre: lustre-OST0001: new disk, initializing [ 4625.811169] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4625.813502] Lustre: 267120:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4627.682713] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4627.842543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4627.848096] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4627.877466] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4632.396455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4633.542506] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4635.203131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4635.207983] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4638.177720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4638.180186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4638.181182] Lustre: Skipped 3 previous similar messages [ 4638.183377] Lustre: Skipped 4 previous similar messages [ 4643.040469] Lustre: server umount lustre-MDT0000 complete [ 4643.296648] LustreError: 264720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.305126] LustreError: 264720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4644.337133] LustreError: 264701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032229 with bad export cookie 16902883791870706478 [ 4644.340346] LustreError: 264701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4644.340358] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4644.466563] Lustre: server umount lustre-MDT0001 complete [ 4655.620759] Lustre: server umount lustre-OST0000 complete [ 4667.040049] Lustre: server umount lustre-OST0001 complete [ 4672.183245] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4675.543051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.677228] LustreError: 270186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4675.695456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4675.697734] Lustre: Skipped 3 previous similar messages [ 4676.944560] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4680.102173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.478542] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4682.771931] Lustre: 271391:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4684.930036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4686.122804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4686.897399] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4690.032730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4691.117218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4691.895781] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4695.354822] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4696.519192] Lustre: 273325:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4698.115058] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4698.117300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4701.055284] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:11:24 (1781032284) [ 4715.999847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4716.002276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.007701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4716.009575] Lustre: Skipped 1 previous similar message [ 4718.683393] Lustre: server umount lustre-MDT0000 complete [ 4719.883224] LustreError: 270168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032304 with bad export cookie 16902883791870712785 [ 4719.887211] LustreError: 270168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4719.887486] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4722.144129] LustreError: 270183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4722.148919] LustreError: 270183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4726.066743] Lustre: server umount lustre-MDT0001 complete [ 4733.437952] Lustre: server umount lustre-OST0000 complete [ 4740.757800] Lustre: server umount lustre-OST0001 complete [ 4742.789930] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4746.044729] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4760.607779] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4763.902614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4763.995857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4764.007529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4764.043817] Lustre: lustre-MDT0000: new disk, initializing [ 4764.074296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4765.404402] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4770.003237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.050386] Lustre: 277506:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4770.173971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4771.588834] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4776.431057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.536455] Lustre: 279201:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4776.789828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4778.421139] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4782.948713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4784.498672] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4785.038341] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4790.126344] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4795.747362] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:12:59 (1781032379) [ 4799.970870] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.972970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4799.975379] Lustre: Skipped 7 previous similar messages [ 4799.977324] Lustre: Skipped 9 previous similar messages [ 4802.683607] Lustre: server umount lustre-MDT0000 complete [ 4804.396951] LustreError: 277497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032389 with bad export cookie 16902883791870720667 [ 4804.401817] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4804.403811] LustreError: 277497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4804.540277] Lustre: server umount lustre-MDT0001 complete [ 4815.173315] Lustre: server umount lustre-OST0000 complete [ 4826.655451] Lustre: server umount lustre-OST0001 complete [ 4829.477641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.624945] LustreError: 282371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4829.630182] LustreError: 282371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4829.657894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.659984] Lustre: Skipped 7 previous similar messages [ 4831.006121] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4834.277299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.895706] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4838.140284] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4839.461105] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4841.989663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4844.165894] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4846.491823] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4864.234081] loop1: detected capacity change from 0 to 204800000 [ 4864.312568] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4864.316140] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4864.320870] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4866.485727] loop1: detected capacity change from 0 to 204800000 [ 4869.064706] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4869.113755] LustreError: 282361:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs18329: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 4869.121585] LustreError: 282361:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs18329 [ 4869.125078] LustreError: 282361:0:(mgs_handler.c:517:mgs_target_reg()) fs18329-MDT0001: Error checking target index: rc = -13 [ 4869.128893] LustreError: fs18329-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4869.136184] LustreError: 285914:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 4869.141284] LustreError: 285914:0:(tgt_mount.c:2084:server_put_super()) no obd fs18329-MDT0001 [ 4869.145286] LustreError: 285914:0:(tgt_mount.c:116:server_deregister_mount()) fs18329-MDT0001 not registered [ 4869.162055] Lustre: server umount fs18329-MDT0001 complete [ 4869.164437] LustreError: 285914:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4872.402546] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.433495] Lustre: Setting parameter fs18329-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs18329-MDT0001 [ 4872.439809] Lustre: Skipped 2 previous similar messages [ 4872.460072] Lustre: srv-fs18329-MDT0001: No data found on store. Initialize space. [ 4872.463013] Lustre: Skipped 4 previous similar messages [ 4872.512292] Lustre: fs18329-MDT0001: new disk, initializing [ 4872.514571] Lustre: Skipped 3 previous similar messages [ 4872.549713] Lustre: srv-fs18329-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4872.553438] Lustre: 286379:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18329-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4872.558079] LustreError: 286379:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18329-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4874.122389] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4878.137422] Lustre: server umount fs18329-MDT0001 complete [ 4883.338745] loop1: detected capacity change from 0 to 204800000 [ 4885.273188] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4885.406715] Lustre: 287975:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18329-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4885.409361] LustreError: 287975:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18329-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4886.618666] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4889.174645] Lustre: server umount fs18329-MDT0001 complete [ 4892.699294] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 4895.714610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4895.715534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.722952] Lustre: Skipped 3 previous similar messages [ 4898.463746] Lustre: server umount lustre-MDT0000 complete [ 4901.188082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4901.228515] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4902.581132] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4905.378523] loop1: detected capacity change from 0 to 204800000 [ 4906.470400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4907.210292] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.334916] Lustre: 290479:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18329-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4907.337450] LustreError: 290479:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18329-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4908.766513] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4910.669373] Lustre: server umount fs18329-MDT0001 complete [ 4921.311804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4923.345120] Lustre: server umount lustre-OST0000 complete [ 4930.710614] Lustre: server umount lustre-MDT0000 complete [ 4931.961296] LustreError: 284094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032516 with bad export cookie 16902883791870723341 [ 4931.964775] LustreError: 284094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4932.084870] Lustre: server umount lustre-MDT0001 complete [ 4935.717077] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4936.734775] Key type lgssc unregistered [ 4936.894641] LNet: 292831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4936.901452] LNetError: 292831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4936.911476] LNet: Removed LNI 192.168.203.150@tcp [ 4937.191102] Key type .llcrypt unregistered [ 4937.192151] Key type ._llcrypt unregistered [ 4944.732790] Key type ._llcrypt registered [ 4944.733757] Key type .llcrypt registered [ 4944.766799] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4945.056334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4945.061841] alg: No test for adler32 (adler32-zlib) [ 4945.904956] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 4945.980043] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4947.559079] Key type lgssc registered [ 4947.910970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4950.350629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4950.357482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.512879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4952.809864] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4955.915285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.049306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4957.287755] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4959.287463] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4960.381704] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4962.520299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4962.641436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4964.411157] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4965.731989] LustreError: 296286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4965.737787] LustreError: 296286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4966.396917] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4970.978232] LustreError: 296286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.099261] LustreError: 295762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.103940] LustreError: 295762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4981.221182] LustreError: 296286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.228654] LustreError: 296286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4982.973804] loop1: detected capacity change from 0 to 204800000 [ 4983.043727] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4983.047838] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4983.050925] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4985.122435] loop1: detected capacity change from 0 to 204800000 [ 4986.341933] LustreError: 295760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4986.349055] LustreError: 295760:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4987.461997] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4987.534668] Lustre: fs21153-OST0000: new disk, initializing [ 4987.537521] Lustre: srv-fs21153-OST0000: No data found on store. Initialize space. [ 4987.542251] Lustre: 297622:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4987.573504] Lustre: fs21153-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.601378] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4993.611700] Lustre: server umount fs21153-OST0000 complete [ 4996.580591] LustreError: 295763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.585618] LustreError: 295763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4998.446806] loop1: detected capacity change from 0 to 204800000 [ 5000.145533] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5000.214582] Lustre: fs21153-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5001.956445] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5004.954206] loop2: detected capacity change from 0 to 204800000 [ 5005.035785] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5005.038969] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5005.042074] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5006.645081] loop2: detected capacity change from 0 to 204800000 [ 5008.372518] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5008.401223] LustreError: fs21153-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 5008.405959] LustreError: fs21153-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5008.408715] LustreError: 300722:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 5008.410837] LustreError: 300722:0:(tgt_mount.c:2084:server_put_super()) no obd fs21153-OST0001 [ 5008.412650] LustreError: 300722:0:(tgt_mount.c:116:server_deregister_mount()) fs21153-OST0001 not registered [ 5008.421805] Lustre: server umount fs21153-OST0001 complete [ 5008.423080] LustreError: 300722:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5011.620453] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5011.660670] Lustre: fs21153-OST0001: new disk, initializing [ 5011.662288] Lustre: srv-fs21153-OST0001: No data found on store. Initialize space. [ 5011.664618] Lustre: 301274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5011.686306] Lustre: fs21153-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5013.440142] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5017.061536] LustreError: 296286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5017.068652] LustreError: 296286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5017.315036] Lustre: server umount fs21153-OST0000 complete [ 5020.685125] Lustre: server umount fs21153-OST0001 complete [ 5024.625482] loop1: detected capacity change from 0 to 204800000 [ 5026.532274] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5028.406030] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5031.466699] loop2: detected capacity change from 0 to 204800000 [ 5033.486674] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5033.577293] Lustre: fs21153-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5033.581419] Lustre: Skipped 1 previous similar message [ 5035.721284] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5039.436740] Lustre: server umount fs21153-OST0000 complete [ 5043.003801] Lustre: server umount fs21153-OST0001 complete [ 5052.898237] LustreError: 295760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.900078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.913818] LustreError: 295760:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 5052.923534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5052.923695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5052.937441] Lustre: Skipped 1 previous similar message [ 5058.016273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5058.020423] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 5062.111123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5062.169831] Lustre: server umount lustre-OST0000 complete [ 5063.647875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5063.650139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5063.653434] Lustre: Skipped 1 previous similar message [ 5063.655181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5063.657031] Lustre: Skipped 1 previous similar message [ 5068.256811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5068.257687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5069.685253] Lustre: server umount lustre-MDT0000 complete [ 5071.027821] LustreError: 294044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032655 with bad export cookie 10240912862243429500 [ 5071.031973] LustreError: 294044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5071.032073] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5071.168909] Lustre: server umount lustre-MDT0001 complete [ 5075.322518] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5076.178975] Key type lgssc unregistered [ 5076.303311] LNet: 307264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5076.305763] LNetError: 307264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5076.313332] LNet: Removed LNI 192.168.203.150@tcp [ 5076.726154] Key type .llcrypt unregistered [ 5076.728316] Key type ._llcrypt unregistered [ 5088.583985] Key type ._llcrypt registered [ 5088.585205] Key type .llcrypt registered [ 5088.631374] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 5095.298678] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5095.624029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5095.644151] alg: No test for adler32 (adler32-zlib) [ 5096.494468] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 5096.579628] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5098.167109] Key type lgssc registered [ 5098.519581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5112.485139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5112.494683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5113.608541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5113.621767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5113.669597] Lustre: lustre-MDT0000: new disk, initializing [ 5113.698822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5113.709323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5115.157697] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5120.476985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5120.544180] Lustre: 311061:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5120.582776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5120.586613] Lustre: Skipped 1 previous similar message [ 5120.635370] Lustre: lustre-MDT0001: new disk, initializing [ 5120.668525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5120.682551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5120.686041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5122.378070] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5126.163951] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5127.779725] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5130.815026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5130.906383] Lustre: lustre-OST0000: new disk, initializing [ 5130.907871] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5130.910352] Lustre: 312899:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5130.937292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5133.339284] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5136.376027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5136.378841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5136.409528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5136.676382] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5137.329620] Lustre: Mounted lustre-client [ 5137.646248] Lustre: Unmounted lustre-client [ 5141.473603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5141.474710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5141.478435] LustreError: Skipped 1 previous similar message [ 5141.489195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5141.495204] Lustre: Skipped 1 previous similar message [ 5142.498347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5142.501682] Lustre: Skipped 1 previous similar message [ 5143.408582] Lustre: server umount lustre-OST0000 complete [ 5146.592374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5146.597843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.606579] Lustre: Skipped 1 previous similar message [ 5146.610319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.618583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.835114] Lustre: server umount lustre-MDT0000 complete [ 5152.258821] LustreError: 311054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032737 with bad export cookie 3820400542332258616 [ 5152.264599] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5152.264987] LustreError: 311054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5152.408623] Lustre: server umount lustre-MDT0001 complete [ 5156.058977] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5157.152558] Key type lgssc unregistered [ 5189.023271] LNet: 314975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5189.025415] LNetError: 314975:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5268.901568] LNet: Removed LNI 192.168.203.150@tcp [ 5269.350359] Key type .llcrypt unregistered [ 5269.352624] Key type ._llcrypt unregistered [ 5276.708792] Key type ._llcrypt registered [ 5276.709980] Key type .llcrypt registered [ 5276.762810] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 5283.576857] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5283.953184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5283.967568] alg: No test for adler32 (adler32-zlib) [ 5284.812077] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 5284.890626] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5286.463186] Key type lgssc registered [ 5286.898060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5298.536183] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:21:21 (1781032881) [ 5301.259084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5301.272366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5302.389261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5302.402668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5302.442757] Lustre: lustre-MDT0000: new disk, initializing [ 5302.502335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5302.512385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5304.274624] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5309.393067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5309.440514] Lustre: 318139:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5309.467489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5309.470763] Lustre: Skipped 1 previous similar message [ 5309.511532] Lustre: lustre-MDT0001: new disk, initializing [ 5309.544589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5309.556481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5309.559339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5310.937161] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5315.310632] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5316.866602] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5319.938045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5320.094272] Lustre: lustre-OST0000: new disk, initializing [ 5320.096933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5320.101048] Lustre: 319977:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5320.141270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.694187] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5323.961344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5323.966311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5324.002356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5326.802478] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5334.497430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5334.497561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5334.499548] LustreError: Skipped 1 previous similar message [ 5334.500154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5334.506674] Lustre: Skipped 1 previous similar message [ 5339.226378] Lustre: server umount lustre-OST0000 complete [ 5340.639817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5340.641802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.645225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5340.646814] Lustre: Skipped 1 previous similar message [ 5344.737142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5344.738308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.602214] Lustre: server umount lustre-MDT0000 complete [ 5348.180828] LustreError: 318967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032933 with bad export cookie 15281272384261594270 [ 5348.188378] LustreError: 318967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5348.188496] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5348.346635] Lustre: server umount lustre-MDT0001 complete [ 5352.752528] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5354.038499] Key type lgssc unregistered [ 5354.180510] LNet: 321801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5354.185394] LNetError: 321801:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5354.202511] LNet: Removed LNI 192.168.203.150@tcp [ 5354.645723] Key type .llcrypt unregistered [ 5354.647819] Key type ._llcrypt unregistered [ 5364.247237] Key type ._llcrypt registered [ 5364.248338] Key type .llcrypt registered [ 5364.284642] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5364.691263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5364.700339] alg: No test for adler32 (adler32-zlib) [ 5365.542868] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 5365.620071] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5367.199170] Key type lgssc registered [ 5367.692857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5370.673091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5370.678347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.858913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5373.233386] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5376.922894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5377.057555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5378.463856] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5380.678976] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5381.910762] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5384.603187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5384.728385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.889461] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5389.346636] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5402.760224] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5403.281829] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5405.217112] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 5406.801369] Lustre: server umount lustre-MDT0000 complete [ 5407.200716] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5407.202623] LustreError: 324309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5407.205301] Lustre: Skipped 1 previous similar message [ 5407.210573] LustreError: 324309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5408.027864] LustreError: 323014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032992 with bad export cookie 13124968882703072918 [ 5408.032513] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5408.034529] LustreError: 323014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5410.159801] Lustre: server umount lustre-MDT0001 complete [ 5435.871100] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5435.933809] Lustre: server umount lustre-OST0000 complete [ 5439.545497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5441.069053] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5442.765231] Lustre: 326972:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.150@tcp [ 5443.785877] Lustre: server umount MGS complete [ 5447.030338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5447.182216] LustreError: 327542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5447.186951] LustreError: 327542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5447.203839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5448.534964] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5451.219134] LustreError: 327539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.343112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5452.484196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5453.934082] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5457.612482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5459.601870] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5462.399206] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5463.169904] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 5464.810585] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5477.180316] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 11 sec [ 5482.468303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5482.469214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5482.482558] Lustre: Skipped 1 previous similar message [ 5482.486747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5482.490771] Lustre: Skipped 1 previous similar message [ 5484.581082] Lustre: server umount lustre-OST0000 complete [ 5487.584716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5487.585265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5487.588850] Lustre: Skipped 2 previous similar messages [ 5492.529455] Lustre: server umount lustre-MDT0000 complete [ 5492.705060] LustreError: 327562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.717838] LustreError: 327562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5494.256889] LustreError: 327524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033079 with bad export cookie 13124968882703073744 [ 5494.263252] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5494.263778] LustreError: 327524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5494.392868] Lustre: server umount lustre-MDT0001 complete [ 5498.623206] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5499.687706] Key type lgssc unregistered [ 5499.839786] LNet: 330882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5499.844207] LNetError: 330882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5499.854593] LNet: Removed LNI 192.168.203.150@tcp [ 5500.300416] Key type .llcrypt unregistered [ 5500.302656] Key type ._llcrypt unregistered [ 5507.951771] Key type ._llcrypt registered [ 5507.953863] Key type .llcrypt registered [ 5508.021253] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 5515.898995] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5516.260118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5516.311368] alg: No test for adler32 (adler32-zlib) [ 5517.179173] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 5517.272856] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5518.863170] Key type lgssc registered [ 5519.410859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5535.338802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5535.354788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.468680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5536.481657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5536.526209] Lustre: lustre-MDT0000: new disk, initializing [ 5536.568211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5536.578390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5538.129536] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5543.603672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.675602] Lustre: 333982:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5543.711929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5543.717112] Lustre: Skipped 1 previous similar message [ 5543.765235] Lustre: lustre-MDT0001: new disk, initializing [ 5543.803805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5543.822156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5543.828451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5545.404173] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5549.456309] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5550.872589] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5553.959638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5554.077232] Lustre: lustre-OST0000: new disk, initializing [ 5554.078729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5554.080893] Lustre: 335817:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5554.105754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5556.002245] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5560.333802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5560.339217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5560.381533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5560.450461] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5562.168226] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5562.266559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5563.686358] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5563.762564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5565.411650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5565.413311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5565.416114] LustreError: Skipped 1 previous similar message [ 5565.424184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5565.429478] Lustre: Skipped 1 previous similar message [ 5566.765078] Lustre: server umount lustre-OST0000 complete [ 5569.504427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5569.509329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.517042] Lustre: Skipped 1 previous similar message [ 5569.520585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5570.529122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5570.530522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5574.513716] Lustre: server umount lustre-MDT0000 complete [ 5575.954182] LustreError: 333974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033160 with bad export cookie 6914599085217918051 [ 5575.961689] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5575.962516] LustreError: 333974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5576.105011] Lustre: server umount lustre-MDT0001 complete [ 5579.973547] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 15:26:03 (1781033163) [ 5586.612965] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 5590.350701] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5606.872906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5607.018176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5607.037304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5607.082027] Lustre: lustre-MDT0000: new disk, initializing [ 5607.107749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.114641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5608.794555] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5614.796905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.864589] Lustre: 340437:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5614.905790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5614.910621] Lustre: Skipped 1 previous similar message [ 5614.958574] Lustre: lustre-MDT0001: new disk, initializing [ 5615.000309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5615.017996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5615.024717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5616.466779] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5620.666125] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5622.071287] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5624.636567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5624.757436] Lustre: lustre-OST0000: new disk, initializing [ 5624.759349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5624.761935] Lustre: 342277:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5624.792107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5627.177549] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5631.247929] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5632.519090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5632.525627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5632.575116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5647.844622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.846815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5647.854844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5647.857914] LustreError: Skipped 1 previous similar message [ 5647.861702] Lustre: Skipped 1 previous similar message [ 5652.962904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5652.965166] Lustre: Skipped 2 previous similar messages [ 5653.546192] Lustre: server umount lustre-OST0000 complete [ 5656.031760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5656.033796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.036769] Lustre: Skipped 1 previous similar message [ 5661.404257] Lustre: server umount lustre-MDT0000 complete [ 5662.729948] LustreError: 340429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033247 with bad export cookie 6914599085217918940 [ 5662.735455] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5662.735697] LustreError: 340429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5662.897517] Lustre: server umount lustre-MDT0001 complete [ 5667.647892] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5669.046283] Key type lgssc unregistered [ 5669.205665] LNet: 344342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5669.210744] LNetError: 344342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5669.220574] LNet: Removed LNI 192.168.203.150@tcp [ 5669.685459] Key type .llcrypt unregistered [ 5669.687832] Key type ._llcrypt unregistered [ 5710.232649] Key type ._llcrypt registered [ 5710.233896] Key type .llcrypt registered [ 5710.281610] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5710.433293] Key type .llcrypt unregistered [ 5710.435418] Key type ._llcrypt unregistered [ 5743.276445] Key type ._llcrypt registered [ 5743.277362] Key type .llcrypt registered [ 5743.310198] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5743.390318] Key type .llcrypt unregistered [ 5743.391255] Key type ._llcrypt unregistered [ 5756.649685] Key type ._llcrypt registered [ 5756.650843] Key type .llcrypt registered [ 5756.692546] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5756.852486] Key type .llcrypt unregistered [ 5756.854635] Key type ._llcrypt unregistered [ 5769.253978] Key type ._llcrypt registered [ 5769.255610] Key type .llcrypt registered [ 5769.292529] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5769.403427] Key type .llcrypt unregistered [ 5769.404684] Key type ._llcrypt unregistered [ 5801.718601] Key type ._llcrypt registered [ 5801.720123] Key type .llcrypt registered [ 5801.777455] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5801.938680] Key type .llcrypt unregistered [ 5801.940902] Key type ._llcrypt unregistered [ 5815.721117] Key type ._llcrypt registered [ 5815.722189] Key type .llcrypt registered [ 5815.760534] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5815.859203] Key type .llcrypt unregistered [ 5815.860754] Key type ._llcrypt unregistered [ 5830.260375] Key type ._llcrypt registered [ 5830.261488] Key type .llcrypt registered [ 5830.302870] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5830.447431] Key type .llcrypt unregistered [ 5830.448738] Key type ._llcrypt unregistered [ 5844.093499] Key type ._llcrypt registered [ 5844.094487] Key type .llcrypt registered [ 5844.130381] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5844.228257] Key type .llcrypt unregistered [ 5844.229394] Key type ._llcrypt unregistered [ 5856.994335] Key type ._llcrypt registered [ 5856.995396] Key type .llcrypt registered [ 5857.029948] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5857.118169] Key type .llcrypt unregistered [ 5857.119192] Key type ._llcrypt unregistered [ 5870.696374] Key type ._llcrypt registered [ 5870.697811] Key type .llcrypt registered [ 5870.747126] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5870.891517] Key type .llcrypt unregistered [ 5870.894627] Key type ._llcrypt unregistered [ 5896.012327] Key type ._llcrypt registered [ 5896.014523] Key type .llcrypt registered [ 5896.061773] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5896.193565] Key type .llcrypt unregistered [ 5896.195584] Key type ._llcrypt unregistered [ 5908.849838] Key type ._llcrypt registered [ 5908.851616] Key type .llcrypt registered [ 5908.893389] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5908.989560] Key type .llcrypt unregistered [ 5908.991455] Key type ._llcrypt unregistered [ 5922.999045] Key type ._llcrypt registered [ 5923.000363] Key type .llcrypt registered [ 5923.051087] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5923.177482] Key type .llcrypt unregistered [ 5923.179655] Key type ._llcrypt unregistered [ 5947.815911] Key type ._llcrypt registered [ 5947.817477] Key type .llcrypt registered [ 5947.875683] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 5955.772758] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5956.103190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5956.150342] alg: No test for adler32 (adler32-zlib) [ 5957.016426] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 5957.113300] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5958.695178] Key type lgssc registered [ 5959.178569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5975.250340] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5975.264513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5976.377727] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5976.387771] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5976.428470] Lustre: tst250fs-MDT0000: new disk, initializing [ 5976.453352] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5976.461607] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5978.033803] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5984.105046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5984.177096] Lustre: 364009:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5984.219707] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5984.225152] Lustre: Skipped 1 previous similar message [ 5984.273649] Lustre: tst250fs-MDT0001: new disk, initializing [ 5984.315834] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5984.333145] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5984.338356] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5985.908707] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5989.262385] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5992.357835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5992.501115] Lustre: tst250fs-OST0000: new disk, initializing [ 5992.504585] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5992.510108] Lustre: 365701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5992.563200] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5993.164923] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5993.171455] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5993.251577] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5995.111476] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6001.114663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6001.224868] Lustre: tst250fs-OST0001: new disk, initializing [ 6001.228403] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 6001.233920] Lustre: 366607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6001.291455] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6003.266518] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6003.270440] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6003.303275] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6003.711888] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6007.466372] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 6020.064584] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6020.069821] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6020.078990] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6023.653370] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.653756] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6023.661219] Lustre: Skipped 2 previous similar messages [ 6023.665893] Lustre: Skipped 2 previous similar messages [ 6025.848142] Lustre: server umount tst250fs-MDT0000 complete [ 6027.630154] LustreError: 366606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033612 with bad export cookie 13869808223716191811 [ 6027.636588] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6027.638029] LustreError: 366606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6027.792910] Lustre: server umount tst250fs-MDT0001 complete [ 6039.894371] Lustre: server umount tst250fs-OST0000 complete [ 6052.023309] Lustre: server umount tst250fs-OST0001 complete [ 6060.411280] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 6064.061147] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6079.031897] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 15:34:22 (1781033662) [ 6079.865680] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 6087.316798] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 6092.195128] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6106.045821] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5822 sec ========== 15:34:49 (1781033689) [ 6106.675978] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:34:50 (1781033690) === [ 6108.184573] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:34:51 (1781033691) === [ 6125.829399] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6127.254114] Key type lgssc unregistered [ 6127.427605] LNet: 375725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6127.432491] LNetError: 375725:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6127.443498] LNet: Removed LNI 192.168.203.150@tcp [ 6127.907164] Key type .llcrypt unregistered [ 6127.910454] Key type ._llcrypt unregistered