[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512584247 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003279] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008371] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010009] pid_max: default: 32768 minimum: 301 [ 0.011126] LSM: Security Framework initializing [ 0.012034] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.022525] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026783] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027136] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030104] *** VALIDATE tmpfs *** [ 0.031468] *** VALIDATE proc *** [ 0.032241] *** VALIDATE cgroup *** [ 0.033025] *** VALIDATE cgroup2 *** [ 0.034261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037025] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041557] debug: unmapping init [mem 0xffffffff93c59000-0xffffffff93c60fff] [ 0.043178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044710] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045021] ... version: 2 [ 0.045952] ... bit width: 48 [ 0.046012] ... generic registers: 4 [ 0.047007] ... value mask: 0000ffffffffffff [ 0.048008] ... max period: 00007fffffffffff [ 0.049009] ... fixed-purpose events: 3 [ 0.050009] ... event mask: 000000070000000f [ 0.052189] rcu: Hierarchical SRCU implementation. [ 0.054108] smp: Bringing up secondary CPUs ... [ 0.055401] x86: Booting SMP configuration: [ 0.056022] .... node #0, CPUs: #1 #2 #3 [ 0.060022] smp: Brought up 1 node, 4 CPUs [ 0.062018] smpboot: Max logical packages: 1 [ 0.063029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.222719] node 0 deferred pages initialised in 156ms [ 0.224140] devtmpfs: initialized [ 0.225239] x86/mm: Memory block size: 128MB [ 0.229436] gcov: version magic: 0x41383552 [ 0.232435] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.235109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.238352] pinctrl core: initialized pinctrl subsystem [ 0.240183] [ 0.240735] ************************************************************* [ 0.242018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.244013] ** ** [ 0.247014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249031] ** ** [ 0.251013] ** This means that this kernel is built to expose internal ** [ 0.253011] ** IOMMU data structures, which may compromise security on ** [ 0.254011] ** your system. ** [ 0.256011] ** ** [ 0.258011] ** If you see this message and you are not debugging the ** [ 0.260011] ** kernel, report this immediately to your vendor! ** [ 0.261009] ** ** [ 0.263012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.265010] ************************************************************* [ 0.266558] NET: Registered protocol family 16 [ 0.268392] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.270058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.272058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.275083] cpuidle: using governor menu [ 0.276624] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.279532] PCI: Using configuration type 1 for base access [ 0.281171] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.289126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.291041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.294090] cryptd: max_cpu_qlen set to 1000 [ 0.297272] ACPI: Added _OSI(Module Device) [ 0.298034] ACPI: Added _OSI(Processor Device) [ 0.300040] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.302036] ACPI: Added _OSI(Processor Aggregator Device) [ 0.308724] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.315328] ACPI: Interpreter enabled [ 0.317100] ACPI: PM: (supports S0 S3 S4 S5) [ 0.319020] ACPI: Using IOAPIC for interrupt routing [ 0.321122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.324408] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.333855] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.336033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.338017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.340079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.345000] acpiphp: Slot [2] registered [ 0.345000] acpiphp: Slot [5] registered [ 0.347115] acpiphp: Slot [6] registered [ 0.348108] acpiphp: Slot [7] registered [ 0.349102] acpiphp: Slot [8] registered [ 0.350122] acpiphp: Slot [9] registered [ 0.351099] acpiphp: Slot [10] registered [ 0.352151] acpiphp: Slot [3] registered [ 0.354102] acpiphp: Slot [4] registered [ 0.355082] acpiphp: Slot [11] registered [ 0.356082] acpiphp: Slot [12] registered [ 0.357101] acpiphp: Slot [13] registered [ 0.358090] acpiphp: Slot [14] registered [ 0.359106] acpiphp: Slot [15] registered [ 0.360088] acpiphp: Slot [16] registered [ 0.361086] acpiphp: Slot [17] registered [ 0.363101] acpiphp: Slot [18] registered [ 0.364102] acpiphp: Slot [19] registered [ 0.365081] acpiphp: Slot [20] registered [ 0.366098] acpiphp: Slot [21] registered [ 0.367074] acpiphp: Slot [22] registered [ 0.367999] acpiphp: Slot [23] registered [ 0.369097] acpiphp: Slot [24] registered [ 0.370069] acpiphp: Slot [25] registered [ 0.371096] acpiphp: Slot [26] registered [ 0.372072] acpiphp: Slot [27] registered [ 0.373079] acpiphp: Slot [28] registered [ 0.374086] acpiphp: Slot [29] registered [ 0.376105] acpiphp: Slot [30] registered [ 0.377112] acpiphp: Slot [31] registered [ 0.379069] PCI host bridge to bus 0000:00 [ 0.380015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.381021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.383019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.385021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.387022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.389024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.390157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.392902] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.395119] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.404017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.408861] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.410020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.412027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.414014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.416336] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.419353] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.421041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.424900] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.429020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.441026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.446977] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.453455] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.460017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.470017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.488022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.500437] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.506019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.524024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.550019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.561913] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.574016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.583020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.616060] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.639020] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.646020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.654020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.671016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.680533] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.689026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.699033] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.721033] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.740392] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.747019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.754022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.767018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.778405] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.780514] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.783581] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.788490] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.791255] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.796211] iommu: Default domain type: Passthrough [ 0.798474] SCSI subsystem initialized [ 0.800138] ACPI: bus type USB registered [ 0.802145] usbcore: registered new interface driver usbfs [ 0.804089] usbcore: registered new interface driver hub [ 0.806096] usbcore: registered new device driver usb [ 0.808212] pps_core: LinuxPPS API ver. 1 registered [ 0.810014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.813076] PTP clock support registered [ 0.815066] EDAC MC: Ver: 3.0.0 [ 0.817242] PCI: Using ACPI for IRQ routing [ 0.819048] NetLabel: Initializing [ 0.821018] NetLabel: domain hash size = 128 [ 0.823029] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.825115] NetLabel: unlabeled traffic allowed by default [ 0.827127] vgaarb: loaded [ 0.828319] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.830015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.836769] clocksource: Switched to clocksource kvm-clock [ 0.946570] VFS: Disk quotas dquot_6.6.0 [ 0.948416] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.951264] *** VALIDATE ramfs *** [ 0.952660] *** VALIDATE hugetlbfs *** [ 0.954471] pnp: PnP ACPI init [ 0.956964] pnp: PnP ACPI: found 6 devices [ 0.976801] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.980501] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.982874] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.984518] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.986681] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.988777] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.991270] NET: Registered protocol family 2 [ 0.993485] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.997685] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.001323] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.007147] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.010962] TCP: Hash tables configured (established 65536 bind 65536) [ 1.014341] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.017140] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.019691] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.022837] NET: Registered protocol family 1 [ 1.025454] RPC: Registered named UNIX socket transport module. [ 1.027869] RPC: Registered udp transport module. [ 1.029728] RPC: Registered tcp transport module. [ 1.031142] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.033782] NET: Registered protocol family 44 [ 1.035496] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.037732] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.040229] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.042863] PCI: CLS 0 bytes, default 64 [ 1.045385] Unpacking initramfs... [ 2.424617] debug: unmapping init [mem 0xffff8d07fcc54000-0xffff8d07fffbffff] [ 2.428162] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.429889] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.431977] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.900497] Initialise system trusted keyrings [ 2.902292] Key type blacklist registered [ 2.904121] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.913224] zbud: loaded [ 2.916109] *** VALIDATE nfs *** [ 2.917347] *** VALIDATE nfs4 *** [ 2.918834] pstore: using deflate compression [ 2.922174] Platform Keyring initialized [ 3.007663] NET: Registered protocol family 38 [ 3.009418] Key type asymmetric registered [ 3.011021] Asymmetric key parser 'x509' registered [ 3.012834] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.015442] io scheduler mq-deadline registered [ 3.016614] io scheduler kyber registered [ 3.017964] io scheduler bfq registered [ 3.019225] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.021277] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.023274] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.025447] ACPI: Power Button [PWRF] [ 3.030538] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.038331] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.051801] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.059326] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.074305] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.100889] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.130217] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.135605] Non-volatile memory driver v1.3 [ 3.136742] Linux agpgart interface v0.103 [ 3.163747] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.165591] vda: detected capacity change from 0 to 74645504 [ 3.176816] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.179724] vdb: detected capacity change from 0 to 1073741824 [ 3.195301] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.198526] vdc: detected capacity change from 0 to 2621440000 [ 3.212785] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.215128] vdd: detected capacity change from 0 to 2621440000 [ 3.226091] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.228175] vde: detected capacity change from 0 to 4294967296 [ 3.239095] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.241175] vdf: detected capacity change from 0 to 4294967296 [ 3.245614] libphy: Fixed MDIO Bus: probed [ 3.254822] usbcore: registered new interface driver usbserial_generic [ 3.256812] usbserial: USB Serial support registered for generic [ 3.258511] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.261925] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.263145] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.265143] mousedev: PS/2 mouse device common for all mice [ 3.268330] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.268515] rtc_cmos 00:05: RTC can wake from S4 [ 3.276055] rtc_cmos 00:05: registered as rtc0 [ 3.278060] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.283100] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.283945] intel_pstate: CPU model not supported [ 3.293199] hid: raw HID events driver (C) Jiri Kosina [ 3.295586] usbcore: registered new interface driver usbhid [ 3.297659] usbhid: USB HID core driver [ 3.299542] drop_monitor: Initializing network drop monitor service [ 3.302280] Initializing XFRM netlink socket [ 3.304093] NET: Registered protocol family 10 [ 3.305148] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.310639] Segment Routing with IPv6 [ 3.312248] NET: Registered protocol family 17 [ 3.314217] mpls_gso: MPLS GSO support [ 3.318555] RAS: Correctable Errors collector initialized. [ 3.321114] AVX version of gcm_enc/dec engaged. [ 3.323134] AES CTR mode by8 optimization enabled [ 3.412707] sched_clock: Marking stable (3412655485, 0)->(4318350029, -905694544) [ 3.415479] registered taskstats version 1 [ 3.416976] Loading compiled-in X.509 certificates [ 3.418466] zswap: loaded using pool lzo/zbud [ 3.444749] Key type big_key registered [ 3.456955] Key type encrypted registered [ 3.458816] ima: No TPM chip found, activating TPM-bypass! [ 3.461193] ima: Allocated hash algorithm: sha1 [ 3.462907] ima: No architecture policies found [ 3.464812] evm: Initialising EVM extended attributes: [ 3.466785] evm: security.selinux [ 3.468029] evm: security.ima [ 3.469249] evm: security.capability [ 3.470758] evm: HMAC attrs: 0x1 [ 3.475271] rtc_cmos 00:05: setting system clock to 2026-07-14 16:08:56 UTC (1784045336) [ 3.482665] debug: unmapping init [mem 0xffffffff94c03000-0xffffffff94dfffff] [ 3.485767] debug: unmapping init [mem 0xffffffff93982000-0xffffffff93c58fff] [ 3.497099] Write protecting the kernel read-only data: 28672k [ 3.500478] debug: unmapping init [mem 0xffffffff92003000-0xffffffff921fffff] [ 3.503417] debug: unmapping init [mem 0xffffffff92914000-0xffffffff929fffff] [ 3.541660] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.550594] systemd[1]: Detected virtualization kvm. [ 3.552490] systemd[1]: Detected architecture x86-64. [ 3.554300] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.582274] systemd[1]: No hostname configured. [ 3.584483] systemd[1]: Set hostname to . [ 3.587239] random: systemd: uninitialized urandom read (16 bytes read) [ 3.590411] systemd[1]: Initializing machine ID from random generator. [ 3.637781] random: ln: uninitialized urandom read (6 bytes read) [ 3.725612] random: systemd: uninitialized urandom read (16 bytes read) [ 3.728364] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.734401] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.738626] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.290699] device-mapper: uevent: version 1.0.3 [ 4.293294] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.028317] virtio_net virtio0 ens2: renamed from eth0 [ 5.072512] random: fast init done [ 5.113966] scsi host0: ata_piix [ 5.151073] scsi host1: ata_piix [ 5.191140] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.193935] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.330537] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.990294] random: crng init done [ 9.991765] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.295688] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.480922] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.701744] SELinux: Disabled at runtime. [ 11.762291] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.771733] systemd[1]: Detected virtualization kvm. [ 11.773397] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.280238] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.284174] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.290051] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.293934] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.296383] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.305145] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.330690] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ 12.476361] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.727941] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.048471] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.071555] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.174787] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.183142] EDAC sbridge: Ver: 1.1.2 [ 14.669080] Key type dns_resolver registered [ 14.959207] NFS: Registering the id_resolver key type [ 14.960746] Key type id_resolver registered [ 14.962414] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg129-server login: [ 67.232128] libcfs: loading out-of-tree module taints kernel. [ 67.267560] Key type ._llcrypt registered [ 67.269746] Key type .llcrypt registered [ 67.446584] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 89.417924] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 91.251397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.262649] alg: No test for adler32 (adler32-zlib) [ 92.839954] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 93.766897] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 95.543240] Key type lgssc registered [ 97.717555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.159280] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.260156] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 176.728430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 176.765679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.000901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.035373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.169420] Lustre: lustre-MDT0000: new disk, initializing [ 178.269378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.286257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.051196] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.563589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.736371] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 197.794722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 197.803882] Lustre: Skipped 1 previous similar message [ 197.949978] Lustre: lustre-MDT0001: new disk, initializing [ 198.152797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.205470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 198.220314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 205.038450] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.634438] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 213.996217] hrtimer: interrupt took 9279851 ns [ 223.661519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.205848] Lustre: lustre-OST0000: new disk, initializing [ 224.208927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.238497] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.390585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.989073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 231.003917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.123620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 233.284479] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.085909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.245889] Lustre: lustre-OST0001: new disk, initializing [ 249.248610] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.255627] Lustre: 9524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.327313] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 256.107421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.119275] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.195272] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.548417] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.841827] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 275.797455] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 281.570925] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing check_logdir /tmp/testlogs/ [ 286.394140] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing yml_node [ 291.269317] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 294.091594] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 297.041707] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 298.850489] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jul 14 12:13:50 EDT 2026 [ 317.750899] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 319.660828] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.400954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.422593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.432600] Lustre: Skipped 2 previous similar messages [ 343.269325] Lustre: server umount lustre-MDT0000 complete [ 347.571962] LustreError: 8448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045680 with bad export cookie 13423680122176198060 [ 347.579661] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.585517] LustreError: 8448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 348.642043] LustreError: 10303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.646055] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.657105] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.657116] Lustre: Skipped 1 previous similar message [ 348.664338] LustreError: 10303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 348.715186] Lustre: Skipped 4 previous similar messages [ 352.672080] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.672312] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.701537] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 352.746448] Lustre: Skipped 2 previous similar messages [ 354.182098] Lustre: server umount lustre-MDT0001 complete [ 358.619359] Lustre: server umount lustre-OST0000 complete [ 363.597402] Lustre: server umount lustre-OST0001 complete [ 370.304978] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 380.009301] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 415.890823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.177283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.211115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.287503] Lustre: lustre-MDT0000: new disk, initializing [ 416.345221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.357847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 420.918949] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 431.907945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.113483] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.209449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.214026] Lustre: Skipped 1 previous similar message [ 432.303354] Lustre: lustre-MDT0001: new disk, initializing [ 432.428129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 432.455385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 432.483143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 437.258215] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 448.238285] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 453.889291] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 461.063199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 461.365757] Lustre: lustre-OST0000: new disk, initializing [ 461.368629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 461.374597] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 461.470811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 463.006630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 463.021631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 463.186231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 469.394966] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 481.320539] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 486.577848] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 486.802415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 492.835376] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 493.129451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 499.175938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 499.179859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.183718] LustreError: Skipped 1 previous similar message [ 499.184559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.217705] Lustre: Skipped 1 previous similar message [ 501.375284] Lustre: server umount lustre-OST0000 complete [ 509.408415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.411574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.417990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 509.418035] Lustre: Skipped 1 previous similar message [ 509.447580] Lustre: Skipped 1 previous similar message [ 511.499071] Lustre: server umount lustre-MDT0000 complete [ 514.533496] LustreError: 17717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.560295] LustreError: 17717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 515.070394] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045848 with bad export cookie 13423680122176199593 [ 515.086421] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 515.087638] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.516643] Lustre: server umount lustre-MDT0001 complete [ 521.948467] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:17:33 (1784045853) [ 542.545144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.878736] LustreError: 19924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.950962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.833114] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 548.322103] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.543209] LustreError: 19924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.566148] LustreError: 19924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 557.751161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.913711] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 571.333965] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 577.767918] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 580.264777] Lustre: Failing over lustre-MDT0000 [ 580.571390] Lustre: server umount lustre-MDT0000 complete [ 583.329815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.342374] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.355715] Lustre: Skipped 1 previous similar message [ 583.371445] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 599.967124] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784045916/real 1784045916] req@ffff8d086053b800 x1870707216780032/t0(0) o400->MGC192.168.201.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784045932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 600.008147] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.967187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.548982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 610.558418] Lustre: Skipped 1 previous similar message [ 610.616913] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 614.886991] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 615.907876] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 615.910944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 615.935382] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 624.573965] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 631.264896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 631.272933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.286020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 631.291622] Lustre: Skipped 1 previous similar message [ 635.228521] Lustre: server umount lustre-MDT0000 complete [ 638.434176] LustreError: 21422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.485770] LustreError: 21422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 639.613803] LustreError: 19906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045972 with bad export cookie 13423680122176201007 [ 639.621358] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.624479] LustreError: 19906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 640.078565] Lustre: server umount lustre-MDT0001 complete [ 650.272083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.021740] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 668.207338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.681253] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 683.239613] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 690.608216] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 692.852768] Lustre: Failing over lustre-MDT0000 [ 692.874720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.892901] Lustre: Skipped 1 previous similar message [ 692.899758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 692.910177] Lustre: Skipped 3 previous similar messages [ 693.089407] Lustre: server umount lustre-MDT0000 complete [ 704.480915] LustreError: 23260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.497314] LustreError: 23260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 709.539369] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784046026/real 1784046026] req@ffff8d0875a95180 x1870707216811648/t0(0) o400->MGC192.168.201.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784046042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 709.553420] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 712.747914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.119311] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 724.304847] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 725.488797] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 725.493632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 725.519087] Lustre: Skipped 1 previous similar message [ 725.534082] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 734.306146] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 739.622278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.634274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 739.657024] Lustre: Skipped 2 previous similar messages [ 739.671611] Lustre: Skipped 1 previous similar message [ 745.010165] Lustre: server umount lustre-MDT0000 complete [ 749.109256] LustreError: 23241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046082 with bad export cookie 13423680122176201945 [ 749.115734] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 749.128580] LustreError: 23241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 749.542260] Lustre: server umount lustre-MDT0001 complete [ 758.717573] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:21:30 (1784046090) [ 776.004195] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 788.272417] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 836.937729] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 850.180086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 850.435048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 850.466535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 850.556053] Lustre: lustre-MDT0000: new disk, initializing [ 850.620880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.628792] Lustre: Skipped 3 previous similar messages [ 850.652226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 856.178804] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 868.374046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.567939] Lustre: 29638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 868.913329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 874.215237] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 885.247443] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 891.460574] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 899.745143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 899.985585] Lustre: 31606:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 901.940198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 902.006475] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 902.092297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 902.114244] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 906.750188] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 916.513692] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 922.932968] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 948.192939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 948.200143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.205812] LustreError: Skipped 1 previous similar message [ 948.207310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 948.235617] Lustre: Skipped 1 previous similar message [ 950.006680] Lustre: server umount lustre-OST0000 complete [ 955.873527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 962.698267] Lustre: server umount lustre-MDT0000 complete [ 963.553712] LustreError: 32328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 963.578674] LustreError: 32328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 966.831690] LustreError: 29628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046299 with bad export cookie 13423680122176202351 [ 966.845945] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 966.848341] LustreError: 29628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 967.438823] Lustre: server umount lustre-MDT0001 complete [ 980.630961] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 984.445765] Key type lgssc unregistered [ 984.834654] LNet: 33585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.838734] LNetError: 33585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.897086] LNet: Removed LNI 192.168.201.129@tcp [ 987.408245] Key type .llcrypt unregistered [ 987.410213] Key type ._llcrypt unregistered [ 1018.137614] Key type ._llcrypt registered [ 1018.141365] Key type .llcrypt registered [ 1018.304221] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1019.844979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.935049] alg: No test for adler32 (adler32-zlib) [ 1021.016624] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 1021.233880] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1022.887955] Key type lgssc registered [ 1024.518673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1034.543734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1034.597444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.200988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.807446] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1046.498522] LustreError: 35005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.512391] LustreError: 35005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1051.624741] LustreError: 35004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.179710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.819639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1057.501153] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1065.379301] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1071.098881] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1078.895528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1079.213398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.302155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1086.458247] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1096.906425] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1097.663612] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1114.117828] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:27:25 (1784046445) [ 1115.971129] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1118.262707] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1126.935631] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:27:37 (1784046457) [ 1136.111783] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:27:47 (1784046467) [ 1137.203291] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1145.825807] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1145.830209] Lustre: Skipped 2 previous similar messages [ 1148.003891] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:27:59 (1784046479) [ 1150.227151] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1150.230721] Lustre: Skipped 2 previous similar messages [ 1158.500556] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1158.506861] Lustre: Skipped 4 previous similar messages [ 1175.309580] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:28:26 (1784046506) [ 1178.548978] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1182.864798] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1193.001819] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:28:44 (1784046524) [ 1195.127741] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1195.132310] Lustre: Skipped 6 previous similar messages [ 1211.348343] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:29:02 (1784046542) [ 1229.802406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1229.811272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.830761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.817637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.820263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.820638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1232.486593] Lustre: server umount lustre-OST0000 complete [ 1236.968872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1236.980923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.001473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.004210] Lustre: Skipped 1 previous similar message [ 1241.057472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.058036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.832834] Lustre: server umount lustre-MDT0000 complete [ 1246.182174] LustreError: 35004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.201223] LustreError: 35004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1246.276550] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046579 with bad export cookie 7897817997573750395 [ 1246.287957] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1246.309504] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1246.671262] Lustre: server umount lustre-MDT0001 complete [ 1257.806439] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1260.273196] Key type lgssc unregistered [ 1260.616970] LNet: 42866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.624799] LNetError: 42866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1260.654652] LNet: Removed LNI 192.168.201.129@tcp [ 1261.474351] Key type .llcrypt unregistered [ 1261.476754] Key type ._llcrypt unregistered [ 1280.688282] Key type ._llcrypt registered [ 1280.690774] Key type .llcrypt registered [ 1280.808712] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 1299.027305] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1300.303627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1300.374474] alg: No test for adler32 (adler32-zlib) [ 1301.436901] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 1301.695262] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1303.377802] Key type lgssc registered [ 1304.897242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1344.505460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1344.527562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.880814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1345.926245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1346.063601] Lustre: lustre-MDT0000: new disk, initializing [ 1346.175875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.205608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1351.873244] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1364.125511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.291578] Lustre: 46021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1364.397256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1364.402255] Lustre: Skipped 1 previous similar message [ 1364.490939] Lustre: lustre-MDT0001: new disk, initializing [ 1364.580567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1364.633813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1364.651882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1369.929847] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1380.776845] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1387.535608] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1395.963687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1396.295862] Lustre: lustre-OST0000: new disk, initializing [ 1396.302324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1396.309555] Lustre: 47992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1396.409427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.863444] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1403.485628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1403.494093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1403.619635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1413.840596] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1419.630893] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1419.833475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1424.484938] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1424.689560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1429.476183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.491304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1429.498700] Lustre: Skipped 1 previous similar message [ 1429.503270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1429.506860] LustreError: Skipped 1 previous similar message [ 1434.594345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1434.607021] Lustre: Skipped 2 previous similar messages [ 1439.715466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1439.717982] Lustre: Skipped 1 previous similar message [ 1441.250229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1441.477157] Lustre: server umount lustre-OST0000 complete [ 1446.380818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1446.391877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.416259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1449.957738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.072281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.089027] Lustre: Skipped 3 previous similar messages [ 1460.191377] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1460.490555] Lustre: server umount lustre-MDT0000 complete [ 1464.784179] LustreError: 46015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046797 with bad export cookie 2314322574552552494 [ 1464.794074] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1464.796589] LustreError: 46015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1465.158281] Lustre: server umount lustre-MDT0001 complete [ 1478.153510] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:33:29 (1784046809) [ 1486.384385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.080646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.450555] LustreError: 50455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.473061] LustreError: 50455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1492.692674] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1497.570319] LustreError: 50456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.702627] LustreError: 50455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.002581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.830683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1510.362727] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1519.141540] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1526.257915] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1533.306049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1533.566629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.078422] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1548.561416] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1564.576065] LustreError: 53010:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1566.029680] Lustre: Modifying parameter general.timeout=1 in log params [ 1567.271346] Lustre: Modifying parameter general.timeout=2 in log params [ 1569.956314] Lustre: Modifying parameter general.timeout=5 in log params [ 1569.958135] Lustre: Skipped 2 previous similar messages [ 1574.177623] Lustre: Modifying parameter general.timeout=9 in log params [ 1574.185140] Lustre: Skipped 3 previous similar messages [ 1582.237781] Lustre: Modifying parameter general.timeout=16 in log params [ 1582.251593] Lustre: Skipped 6 previous similar messages [ 1598.917751] Lustre: Modifying parameter general.timeout=29 in log params [ 1598.926098] Lustre: Skipped 12 previous similar messages [ 1631.785986] Lustre: Modifying parameter general.timeout=58 in log params [ 1631.791975] Lustre: Skipped 28 previous similar messages [ 1695.790633] Lustre: Modifying parameter general.timeout=120 in log params [ 1695.795881] Lustre: Skipped 61 previous similar messages [ 1725.481382] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:37:36 (1784047056) [ 1729.444982] Lustre: 59968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1731.456330] Lustre: 60067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1733.548964] Lustre: 60180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1738.378619] Lustre: 60385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1738.397973] Lustre: 60385:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1774.120522] Lustre: 61850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1774.137757] Lustre: 61850:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1774.192252] LustreError: 61850:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1780.310540] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:38:31 (1784047111) [ 1801.697497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1801.717309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.781417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.792546] Lustre: Skipped 2 previous similar messages [ 1805.042803] Lustre: server umount lustre-MDT0000 complete [ 1808.352880] LustreError: 50456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1808.389064] LustreError: 50456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1808.987161] LustreError: 50437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047142 with bad export cookie 2314322574552553369 [ 1808.989606] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1808.996902] LustreError: 50437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1809.360447] Lustre: server umount lustre-MDT0001 complete [ 1824.130356] Lustre: server umount lustre-OST0000 complete [ 1861.163503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.540824] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1861.595818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1861.602130] Lustre: Skipped 32 previous similar messages [ 1862.111699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.762851] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1877.475494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.632911] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1877.650882] Lustre: 64386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1884.274937] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1893.400712] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1900.155751] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1907.085859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1907.221518] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1907.561495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.566851] Lustre: Skipped 1 previous similar message [ 1914.517301] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1923.545157] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1925.983377] Lustre: 66956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1938.402962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1938.424067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.438983] Lustre: Skipped 2 previous similar messages [ 1938.443230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1938.455935] Lustre: Skipped 3 previous similar messages [ 1940.520849] Lustre: server umount lustre-OST0000 complete [ 1944.544368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1944.551757] LustreError: Skipped 1 previous similar message [ 1949.154181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.173222] Lustre: Skipped 2 previous similar messages [ 1951.141487] Lustre: server umount lustre-MDT0000 complete [ 1954.272538] LustreError: 64398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1954.297031] LustreError: 64398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1955.451955] LustreError: 64381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047288 with bad export cookie 2314322574552556652 [ 1955.473850] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1955.475245] LustreError: 64381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1955.853866] Lustre: server umount lustre-MDT0001 complete [ 1969.704870] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1973.668457] Key type lgssc unregistered [ 1974.106600] LNet: 68166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.115050] LNetError: 68166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.137274] LNet: Removed LNI 192.168.201.129@tcp [ 1975.043215] Key type .llcrypt unregistered [ 1975.045251] Key type ._llcrypt unregistered [ 2006.977532] Key type ._llcrypt registered [ 2006.980311] Key type .llcrypt registered [ 2007.100980] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2008.242071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2008.499845] alg: No test for adler32 (adler32-zlib) [ 2009.625963] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 2010.047821] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2011.839916] Key type lgssc registered [ 2013.509612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2022.540329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2022.586923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.272657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.986049] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2034.674713] LustreError: 69591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.696593] LustreError: 69591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2038.105094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.682586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2044.189603] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2053.588247] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2060.843933] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2069.524467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2070.495878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.029368] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2088.625240] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2103.446971] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2115.554268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2115.561971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.584525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2119.655333] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.663505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2119.679481] Lustre: Skipped 1 previous similar message [ 2119.695346] Lustre: Skipped 2 previous similar messages [ 2121.164848] Lustre: server umount lustre-MDT0000 complete [ 2124.783204] LustreError: 69590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.831245] LustreError: 69590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2126.689843] LustreError: 70330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047459 with bad export cookie 14212423630510898295 [ 2126.704754] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2126.707879] LustreError: 70330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2127.543450] Lustre: server umount lustre-MDT0001 complete [ 2143.383284] Lustre: server umount lustre-OST0000 complete [ 2181.404980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.617456] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2181.660176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2181.937605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.303920] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2198.389166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.542032] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2198.571525] Lustre: 74480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2198.983803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.542622] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2213.134763] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2219.708963] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2227.991958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2228.158057] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2228.723882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.016255] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2244.791817] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2246.924538] Lustre: 77057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2248.963679] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2258.401303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2258.404798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.431439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2261.088681] Lustre: server umount lustre-OST0000 complete [ 2265.574574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2265.581353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.590236] Lustre: Skipped 1 previous similar message [ 2265.597958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2265.603441] Lustre: Skipped 1 previous similar message [ 2271.415572] Lustre: server umount lustre-MDT0000 complete [ 2274.274641] LustreError: 74489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.303953] LustreError: 74489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2275.792954] LustreError: 77059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047608 with bad export cookie 14212423630510899142 [ 2275.794527] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2275.807330] LustreError: 77059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2276.142153] Lustre: server umount lustre-MDT0001 complete [ 2289.579334] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2292.824643] Key type lgssc unregistered [ 2293.123744] LNet: 78274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.130441] LNetError: 78274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.148062] LNet: Removed LNI 192.168.201.129@tcp [ 2294.272708] Key type .llcrypt unregistered [ 2294.275117] Key type ._llcrypt unregistered [ 2322.988519] Key type ._llcrypt registered [ 2322.991413] Key type .llcrypt registered [ 2323.100303] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2324.760538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2324.798595] alg: No test for adler32 (adler32-zlib) [ 2326.224966] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 2326.663174] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2328.431270] Key type lgssc registered [ 2329.754931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2339.963560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2339.991127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.940735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.779940] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2351.075347] LustreError: 79694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.106391] LustreError: 79694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2356.196679] LustreError: 79695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.313409] LustreError: 79694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.653478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.675433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2369.510662] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2378.721079] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2386.689663] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2396.621918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2397.106118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.601546] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2413.494732] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2420.659814] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2422.266993] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2423.698492] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2426.184177] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2426.188355] Lustre: Skipped 1 previous similar message [ 2431.302041] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2431.308365] Lustre: Skipped 3 previous similar messages [ 2439.420227] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2439.430973] Lustre: Skipped 5 previous similar messages [ 2456.034336] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2456.043807] Lustre: Skipped 12 previous similar messages [ 2489.274488] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2489.281995] Lustre: Skipped 23 previous similar messages [ 2552.760909] Lustre: 87813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2554.018661] Lustre: Disabling parameter general.jobid_var= in log params [ 2554.025412] Lustre: Skipped 48 previous similar messages [ 2556.656147] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 12:51:27 (1784047887) [ 2558.772996] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2561.607697] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 12:51:32 (1784047892) [ 2570.220889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.234788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.245600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2572.978727] Lustre: server umount lustre-OST0000 complete [ 2577.379958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2577.384745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.394491] Lustre: Skipped 1 previous similar message [ 2577.400622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2577.406407] Lustre: Skipped 1 previous similar message [ 2580.449814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2580.453643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2580.470726] Lustre: Skipped 1 previous similar message [ 2583.575099] Lustre: server umount lustre-MDT0000 complete [ 2585.577710] LustreError: 79691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.598401] LustreError: 79691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2587.749410] LustreError: 79674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047920 with bad export cookie 7922518951007853781 [ 2587.760459] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2587.765731] LustreError: 79674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2588.055028] Lustre: server umount lustre-MDT0001 complete [ 2601.755565] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2605.100320] Key type lgssc unregistered [ 2605.530764] LNet: 89063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2605.561682] LNetError: 89063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.590324] LNet: Removed LNI 192.168.201.129@tcp [ 2606.657191] Key type .llcrypt unregistered [ 2606.658944] Key type ._llcrypt unregistered [ 2612.568429] Key type ._llcrypt registered [ 2612.574351] Key type .llcrypt registered [ 2612.781453] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2618.706254] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules [ 2619.843850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2619.874172] alg: No test for adler32 (adler32-zlib) [ 2621.062620] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 2621.285198] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2623.023897] Key type lgssc registered [ 2624.340805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2625.524856] LustreError: 90327:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2648.615336] LustreError: 90327:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2649.266585] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2659.722738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2659.743069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.299036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.126818] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2671.594587] LustreError: 91354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.609768] LustreError: 91354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2676.734351] LustreError: 91353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.879420] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:53:28 (1784048008) [ 2681.826720] LustreError: 91354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.971671] LustreError: 91385:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2681.987561] LustreError: 91385:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2682.001299] LustreError: 91385:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2682.715659] Lustre: server umount lustre-MDT0000 complete [ 2704.068377] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2708.428943] Key type lgssc unregistered [ 2708.730696] LNet: 92878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2708.738782] LNetError: 92878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2708.763311] LNet: Removed LNI 192.168.201.129@tcp [ 2709.756236] Key type .llcrypt unregistered [ 2709.761974] Key type ._llcrypt unregistered [ 2733.344791] Key type ._llcrypt registered [ 2733.350828] Key type .llcrypt registered [ 2733.525610] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2734.937043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2734.991748] alg: No test for adler32 (adler32-zlib) [ 2736.229347] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 2736.570882] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2738.263176] Key type lgssc registered [ 2739.671165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2748.484974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2748.502697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2750.222511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.839444] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2760.673192] LustreError: 94101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.691429] LustreError: 94101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2761.711362] LustreError: 94102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.369820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.073610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2772.279072] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2781.711960] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2789.516421] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2799.210512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2800.248440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2807.994755] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2818.121803] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2883.113188] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:56:54 (1784048214) [ 2885.339933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2887.566810] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:56:58 (1784048218) [ 2893.798130] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.807611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.825608] Lustre: Skipped 2 previous similar messages [ 2893.845262] Lustre: Skipped 2 previous similar messages [ 2898.915646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2898.930774] Lustre: Skipped 2 previous similar messages [ 2899.995440] Lustre: server umount lustre-MDT0000 complete [ 2904.034572] LustreError: 94832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.060812] LustreError: 94832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2905.562480] LustreError: 94080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048238 with bad export cookie 14962679152845779060 [ 2905.564107] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2905.590321] LustreError: 94080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2906.114398] Lustre: server umount lustre-MDT0001 complete [ 2922.079329] Lustre: server umount lustre-OST0000 complete [ 2936.794124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.544484] LustreError: 98423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.815244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.956101] LustreError: 98424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2944.482134] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2952.161488] LustreError: 98423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2952.178968] LustreError: 98423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2955.939816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.689775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2962.138621] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2972.064537] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2979.168512] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2995.742683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2996.031727] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2996.058023] LustreError: 98413:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2996.114319] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2996.124418] LustreError: 100510:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2996.131575] LustreError: 100510:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2996.137797] LustreError: 100510:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2996.217746] Lustre: server umount lustre-OST0000 complete [ 2996.219868] LustreError: 100510:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3006.316745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3006.547929] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3006.552732] LustreError: 98411:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3006.595746] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3006.603812] LustreError: 100974:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3006.613084] LustreError: 100974:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3006.623354] LustreError: 100974:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3006.697782] Lustre: server umount lustre-OST0000 complete [ 3006.704644] LustreError: 100974:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3019.705953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3020.038425] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3020.045397] Lustre: Found index 0 for lustre-OST0000, updating log [ 3020.058905] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3020.208651] Lustre: lustre-OST0000: new disk, initializing [ 3020.211773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3020.222938] Lustre: 101509:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3020.785254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3028.021807] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3032.582973] Lustre: Failing over lustre-OST0000 [ 3033.056729] Lustre: server umount lustre-OST0000 complete [ 3038.182770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3038.185468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.206972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.276195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.285390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.393349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3047.396246] Lustre: Skipped 2 previous similar messages [ 3048.549473] Lustre: server umount lustre-MDT0000 complete [ 3052.520055] LustreError: 99534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.565688] LustreError: 99534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3053.600082] Lustre: 93414:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048370/real 1784048370] req@000000004c36d418 x1870709988111104/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784048386 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3053.630206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.772216] LustreError: 98404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048386 with bad export cookie 14962679152845781797 [ 3053.790131] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3054.132627] Lustre: server umount lustre-MDT0001 complete [ 3066.206500] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:59:57 (1784048397) [ 3088.616411] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3100.804677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.105841] LustreError: 103857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3101.125298] LustreError: 103857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3101.219517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.030790] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3118.300234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.875593] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3129.540269] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3139.659548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3140.238848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.249858] Lustre: Skipped 1 previous similar message [ 3140.263453] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3143.355591] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3146.481210] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3146.486211] Lustre: lustre-OST0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3146.515087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3148.177992] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3159.977084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3160.119628] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3160.251495] Lustre: lustre-OST0001: new disk, initializing [ 3160.258141] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3160.278586] Lustre: 106190:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3165.791353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3165.809025] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3165.870652] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3168.561787] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3178.651267] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3185.438321] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3191.265371] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3191.278267] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.308844] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3197.888496] Lustre: server umount lustre-MDT0001 complete [ 3198.095706] LustreError: 106669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.114142] LustreError: 106669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3210.802657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.945843] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3210.951708] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3211.075571] Lustre: 103847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3212.248641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3212.256520] Lustre: Skipped 1 previous similar message [ 3212.665322] LustreError: 107722:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3212.677578] LustreError: 107722:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3212.692926] LustreError: 103849:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3212.813092] LustreError: 107735:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.129@tcp: cfg command failed: rc = -17 [ 3212.821062] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3212.821062] [ 3217.403202] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3217.418927] Lustre: Skipped 1 previous similar message [ 3217.750362] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3229.338309] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:02:40 (1784048560) [ 3259.381143] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3286.511586] Lustre: 109832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3316.215632] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3320.654404] Lustre: 110967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3426.785321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3426.797528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.808169] Lustre: server umount lustre-MDT0000 complete [ 3426.821466] Lustre: Skipped 2 previous similar messages [ 3426.852971] LustreError: 103857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.895175] LustreError: 103857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3431.407528] LustreError: 103837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048764 with bad export cookie 14962679152845782812 [ 3431.409086] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3431.431160] LustreError: 103837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3431.977176] Lustre: server umount lustre-MDT0001 complete [ 3446.635882] Lustre: server umount lustre-OST0000 complete [ 3448.479116] Lustre: 93413:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048765/real 1784048765] req@ffff8d0752339880 x1870709988393600/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784048781 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3448.531971] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.556089] Lustre: Skipped 3 previous similar messages [ 3452.632786] Lustre: server umount lustre-OST0001 complete [ 3460.012923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3473.283424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3483.257586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3494.566472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3519.069657] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3533.767022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.800432] Lustre: lustre-MDT0000: reset Object Index mappings [ 3534.493892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3539.574907] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3548.651417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.681988] Lustre: lustre-MDT0001: reset Object Index mappings [ 3554.857803] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3558.598273] Lustre: 116081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3566.695159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3574.543083] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3574.856968] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 3580.932064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3583.624524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3589.631970] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3589.667131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3590.586684] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3599.138234] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3603.481726] Lustre: 117959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3663.178943] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:09:54 (1784048994) [ 3692.022386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.023847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3692.034210] Lustre: Skipped 2 previous similar messages [ 3692.049671] Lustre: Skipped 8 previous similar messages [ 3704.287295] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3704.628295] Lustre: server umount lustre-MDT0000 complete [ 3707.375557] LustreError: 114922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3707.400211] LustreError: 114922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3709.355193] LustreError: 116464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049042 with bad export cookie 14962679152845829929 [ 3709.356372] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3709.373242] LustreError: 116464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3716.201618] Lustre: server umount lustre-MDT0001 complete [ 3727.379520] Lustre: server umount lustre-OST0000 complete [ 3746.271150] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3746.706424] Lustre: server umount lustre-OST0001 complete [ 3757.613148] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 3770.561450] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3816.956279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.252412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3817.279665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3817.438883] Lustre: lustre-MDT0000: new disk, initializing [ 3817.528948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3817.539413] Lustre: Skipped 3 previous similar messages [ 3817.562854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3823.252383] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3831.085443] Lustre: server umount lustre-MDT0000 complete [ 3842.138338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.675790] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3842.732285] Lustre: 122233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3842.765142] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3848.734777] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3855.258797] Lustre: server umount lustre-MDT0000 complete [ 3866.935214] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:13:18 (1784049198) [ 3869.251402] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3871.819067] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:13:22 (1784049202) [ 3896.006139] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3908.523765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.352984] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3925.218508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.410447] Lustre: 124306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3925.484646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3925.489078] Lustre: Skipped 1 previous similar message [ 3925.617529] Lustre: lustre-MDT0001: new disk, initializing [ 3925.872648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3925.884392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3931.534420] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3936.822969] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3945.275743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3945.591954] Lustre: lustre-OST0000: new disk, initializing [ 3945.595544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3945.601649] Lustre: 125880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3946.975930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3946.988806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3947.136863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3952.798223] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3965.709041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3965.924761] Lustre: lustre-OST0001: new disk, initializing [ 3965.930366] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3965.938973] Lustre: 126753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3967.952856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3967.967634] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3968.125207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3973.580349] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3984.695657] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3989.885991] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4422.277559] Lustre: 133175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4422.286921] Lustre: 133175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4422.294243] Lustre: 133175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4422.301288] Lustre: 133175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4422.308156] Lustre: 133175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4422.314379] Lustre: 133175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4752.851545] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:28:03 (1784050083) [ 4770.784396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4770.793255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4770.800355] Lustre: Skipped 3 previous similar messages [ 4770.805253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4770.810489] Lustre: Skipped 12 previous similar messages [ 4773.904534] Lustre: server umount lustre-MDT0000 complete [ 4776.931130] LustreError: 125951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4776.954833] LustreError: 125951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4777.018673] LustreError: 124296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784050110 with bad export cookie 14962679152845899831 [ 4777.021180] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4777.033188] LustreError: 124296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4782.048771] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.064075] Lustre: Skipped 3 previous similar messages [ 4782.072665] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4782.075472] Lustre: Skipped 5 previous similar messages [ 4783.596531] Lustre: server umount lustre-MDT0001 complete [ 4793.594503] Lustre: server umount lustre-OST0000 complete [ 4803.923822] Lustre: server umount lustre-OST0001 complete [ 4813.819201] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 4825.842342] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 4864.312918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.581301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4864.636338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4864.749691] Lustre: lustre-MDT0000: new disk, initializing [ 4864.847234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4864.854232] Lustre: Skipped 5 previous similar messages [ 4864.872978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4869.554945] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4880.491885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4880.610617] Lustre: 136238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4880.659194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4880.662808] Lustre: Skipped 1 previous similar message [ 4880.736035] Lustre: lustre-MDT0001: new disk, initializing [ 4880.875850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4880.889121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4885.130622] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4895.170405] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4900.630429] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4908.578730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4909.024138] Lustre: lustre-OST0000: new disk, initializing [ 4909.032059] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4909.040970] Lustre: 138205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4910.528141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4910.539140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4910.651292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4916.465501] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4925.390593] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4936.397882] Lustre: lustre-MDD0000: changelog on [ 4939.675360] Lustre: lustre-MDD0001: changelog on [ 6922.367993] Lustre: lustre-MDD0001: changelog off [ 6925.544695] Lustre: 136243:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6926.726834] Lustre: lustre-MDD0000: changelog off [ 6936.057326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.066728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6936.079386] Lustre: Skipped 2 previous similar messages [ 6936.103410] Lustre: Skipped 1 previous similar message [ 6941.159455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6941.171184] Lustre: Skipped 1 previous similar message [ 6946.273995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6946.285889] Lustre: Skipped 1 previous similar message [ 6949.343434] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6949.658846] Lustre: server umount lustre-OST0000 complete [ 6954.465297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6954.472375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6954.480679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6960.894713] Lustre: server umount lustre-MDT0000 complete [ 6961.639435] LustreError: 137399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6961.664210] LustreError: 137399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6965.301527] LustreError: 146324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052298 with bad export cookie 14962679152846727952 [ 6965.314250] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6965.319311] LustreError: 146324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6965.795284] Lustre: server umount lustre-MDT0001 complete [ 6979.512044] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6982.497913] Key type lgssc unregistered [ 6982.835799] LNet: 147367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6982.839491] LNetError: 147367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6982.872975] LNet: Removed LNI 192.168.201.129@tcp [ 6983.868245] Key type .llcrypt unregistered [ 6983.871738] Key type ._llcrypt unregistered [ 7005.869167] Key type ._llcrypt registered [ 7005.873914] Key type .llcrypt registered [ 7005.974355] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 7025.739102] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7027.058949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7027.202053] alg: No test for adler32 (adler32-zlib) [ 7028.382261] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 7028.696651] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 7030.439266] Key type lgssc registered [ 7032.023032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7069.126859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7069.226535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.469474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7070.529734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7070.675079] Lustre: lustre-MDT0000: new disk, initializing [ 7070.801338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.816635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7075.872977] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7089.100386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.257073] Lustre: 150568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7089.403864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7089.408538] Lustre: Skipped 1 previous similar message [ 7089.523705] Lustre: lustre-MDT0001: new disk, initializing [ 7089.694810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7089.756455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7089.775742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7095.095507] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7105.496593] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7111.327548] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7118.362342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7118.641943] Lustre: lustre-OST0000: new disk, initializing [ 7118.645613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7118.655648] Lustre: 152537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7118.761262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7125.112666] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7128.721143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7128.735746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7128.933183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7136.618257] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7142.967843] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_module obdecho/obdecho [ 7145.094130] LustreError: 153544:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7145.105162] LustreError: 153544:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7146.658715] LustreError: 153640:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7159.785361] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7159.786684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7159.803209] Lustre: Skipped 2 previous similar messages [ 7159.820112] Lustre: Skipped 2 previous similar messages [ 7164.909737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7164.921681] Lustre: Skipped 2 previous similar messages [ 7165.164481] Lustre: server umount lustre-MDT0000 complete [ 7169.077360] LustreError: 150562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052502 with bad export cookie 16255254582408975983 [ 7169.091378] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7169.104235] LustreError: 150562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7169.583803] Lustre: server umount lustre-MDT0001 complete [ 7184.114852] Lustre: server umount lustre-OST0000 complete [ 7205.592636] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 7215.707401] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7255.009343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.272593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7255.311137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7255.383448] Lustre: lustre-MDT0000: new disk, initializing [ 7255.490590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7255.502683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7261.552682] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7275.353842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7275.482628] Lustre: 157356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7275.568815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7275.575643] Lustre: Skipped 1 previous similar message [ 7275.674514] Lustre: lustre-MDT0001: new disk, initializing [ 7275.757485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7275.781313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7275.799713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7281.275701] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7291.147476] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7297.981954] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7305.160521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7305.427470] Lustre: lustre-OST0000: new disk, initializing [ 7305.431168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7305.437085] Lustre: 159325:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7305.526899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7312.981843] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7315.573759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7315.591866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7315.760905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7323.951278] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7329.018817] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7329.399477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7335.853601] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7336.152583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7341.536730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7341.538193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7341.553101] LustreError: Skipped 1 previous similar message [ 7341.560322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7341.579503] Lustre: Skipped 1 previous similar message [ 7341.609265] Lustre: Skipped 1 previous similar message [ 7344.764272] Lustre: server umount lustre-OST0000 complete [ 7351.791971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.800724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7351.807525] Lustre: Skipped 1 previous similar message [ 7351.825056] Lustre: Skipped 1 previous similar message [ 7355.686908] Lustre: server umount lustre-MDT0000 complete [ 7359.486088] LustreError: 158151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052692 with bad export cookie 16255254582408976942 [ 7359.487365] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7359.492783] LustreError: 158151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7359.792238] Lustre: server umount lustre-MDT0001 complete [ 7363.555993] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:11:35 (1784052695) [ 7365.523694] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7367.789233] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:11:39 (1784052699) [ 7375.769191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7376.110404] LustreError: 161738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7376.132866] LustreError: 161738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7376.188023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7380.426848] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7381.487872] LustreError: 161739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7386.593578] LustreError: 161738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7388.837802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.536113] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7402.213460] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7408.116290] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7415.330972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7415.760933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.768498] Lustre: Skipped 1 previous similar message [ 7421.756954] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7429.385585] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7440.573099] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.29@tcp (stopping) [ 7441.288993] Lustre: server umount lustre-MDT0000 complete [ 7444.455788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7444.472431] Lustre: Skipped 2 previous similar messages [ 7444.475710] LustreError: 161738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7444.504854] LustreError: 161738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7445.096644] LustreError: 162472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052778 with bad export cookie 16255254582408977824 [ 7445.100359] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7445.106548] LustreError: 162472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7449.568665] LustreError: 161733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.571829] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7449.593568] LustreError: 161733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7449.612761] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7449.612769] Lustre: Skipped 1 previous similar message [ 7451.671669] Lustre: server umount lustre-MDT0001 complete [ 7458.435396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7470.587960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.847790] LustreError: 165583:0:(llog.c:1655:llog_backup()) MGC192.168.201.129@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7470.854980] Lustre: 165583:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.129@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7475.171855] LustreError: 165589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7475.179270] LustreError: 165589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7491.746078] LustreError: 165589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.771261] LustreError: 165589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7511.197507] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.29@tcp (not set up) [ 7511.211113] Lustre: Skipped 2 previous similar messages [ 7525.861109] LustreError: 163572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.879227] LustreError: 163572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 7531.489108] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7531.493751] Lustre: 165583:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.129@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7572.484601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7577.184462] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7577.578976] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7586.263306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7590.881853] LustreError: 148791:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a8b96b13 x1870714488619904/t0(0) o250->MGC192.168.201.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7590.917347] LustreError: 165607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7590.932677] LustreError: 165607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 7591.057834] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.29@tcp (not set up) [ 7595.082208] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7596.530633] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7601.658406] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7603.153304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7608.504393] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7609.791275] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7625.872050] Lustre: server umount lustre-MDT0000 complete [ 7627.238972] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7627.242690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7628.836925] LustreError: 161718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052961 with bad export cookie 16255254582408984278 [ 7628.837501] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7628.845393] LustreError: 161718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7629.066819] Lustre: server umount lustre-MDT0001 complete [ 7643.029652] Lustre: server umount lustre-OST0000 complete [ 7662.002787] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 7672.445163] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7710.750051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7711.015883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7711.050087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7711.119653] Lustre: lustre-MDT0000: new disk, initializing [ 7711.187650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7711.194705] Lustre: Skipped 1 previous similar message [ 7711.217666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7716.058949] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7726.441361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7726.563555] Lustre: 170683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7726.803717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7732.097184] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7743.005585] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7749.925671] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7757.571513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7757.843610] Lustre: lustre-OST0000: new disk, initializing [ 7757.846584] Lustre: Skipped 1 previous similar message [ 7757.849827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7757.859896] Lustre: Skipped 2 previous similar messages [ 7757.869206] Lustre: 172652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7764.168395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7764.180690] Lustre: Skipped 1 previous similar message [ 7764.193786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7764.370714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7766.488885] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7776.852211] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7782.129899] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7782.558914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7788.299525] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7788.508225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7795.199371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7795.203257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.215545] LustreError: Skipped 1 previous similar message [ 7795.216645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7795.246041] Lustre: Skipped 3 previous similar messages [ 7797.160955] Lustre: server umount lustre-OST0000 complete [ 7803.872553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7807.921159] Lustre: server umount lustre-MDT0000 complete [ 7810.536268] LustreError: 170695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.571881] LustreError: 170695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7812.545261] LustreError: 170675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053145 with bad export cookie 16255254582409001113 [ 7812.555234] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7812.562951] LustreError: 170675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7813.134419] Lustre: server umount lustre-MDT0001 complete [ 7824.451592] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:19:15 (1784053155) [ 7831.701065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7836.398435] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7846.259280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.992979] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7860.270619] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7868.368625] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7875.616413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7884.366871] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7893.476976] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7906.316935] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:20:37 (1784053237) [ 7911.906066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.915251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7911.934739] Lustre: Skipped 2 previous similar messages [ 7917.312743] Lustre: server umount lustre-OST0000 complete [ 7927.266370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7927.273566] Lustre: Skipped 9 previous similar messages [ 7928.091716] Lustre: server umount lustre-MDT0000 complete [ 7933.919827] LustreError: 175045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053266 with bad export cookie 16255254582409001995 [ 7933.944045] LustreError: 175045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7933.945110] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7934.519045] Lustre: server umount lustre-MDT0001 complete [ 7948.646266] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 7951.301631] Key type lgssc unregistered [ 7951.611813] LNet: 178796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7951.618960] LNetError: 178796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7951.635284] LNet: Removed LNI 192.168.201.129@tcp [ 7952.342377] Key type .llcrypt unregistered [ 7952.344708] Key type ._llcrypt unregistered [ 7971.530846] Key type ._llcrypt registered [ 7971.532677] Key type .llcrypt registered [ 7971.678602] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7972.939893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7972.957421] alg: No test for adler32 (adler32-zlib) [ 7974.070833] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 7974.439859] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 7976.159396] Key type lgssc registered [ 7977.421773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7985.483144] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7985.505132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8001.055626] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8006.176516] LustreError: 180039:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 8011.295620] LustreError: 180039:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8011.310038] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8011.319588] LustreError: MGC192.168.201.129@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8011.335513] LustreError: 180039:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8011.352854] LustreError: 180039:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8011.376814] LustreError: 180039:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8011.507383] Lustre: server umount lustre-OST0000 complete [ 8011.509353] LustreError: 180039:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8020.731859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8022.162601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.961751] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8032.739874] LustreError: 180539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8032.768188] LustreError: 180539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8032.911242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8033.216849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8037.385989] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8044.032077] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8049.586385] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8055.615951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8055.798751] LustreError: 182355:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8055.817854] Lustre: 182355:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.129@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8055.954950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8061.759336] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8070.249537] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8074.210909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8074.221053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8074.242134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8077.793128] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.798748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8077.836542] Lustre: Skipped 2 previous similar messages [ 8079.786633] Lustre: server umount lustre-MDT0000 complete [ 8082.917090] LustreError: 181588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8082.948849] LustreError: 181588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8084.887508] LustreError: 180518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053417 with bad export cookie 12121115984938757348 [ 8084.904784] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8084.908062] LustreError: 180518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8085.521544] Lustre: server umount lustre-MDT0001 complete [ 8100.941238] Lustre: server umount lustre-OST0000 complete [ 8110.293524] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 8121.393528] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 8160.289254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8160.694325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8160.745824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8160.872189] Lustre: lustre-MDT0000: new disk, initializing [ 8161.015984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8161.029245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8166.438256] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8177.767414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8177.919894] Lustre: 185878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8177.999178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8178.006447] Lustre: Skipped 1 previous similar message [ 8178.121482] Lustre: lustre-MDT0001: new disk, initializing [ 8178.211825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8178.267489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8178.282961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8183.689891] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8193.603704] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8199.658726] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8206.667488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8206.990715] Lustre: lustre-OST0000: new disk, initializing [ 8207.002153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8207.020607] Lustre: 187850:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8207.142972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8213.298421] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8217.193900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8217.206206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8217.279330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8222.907448] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8227.755475] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8228.085521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8232.550311] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8232.743952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8238.051524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8238.052879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8238.076611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8238.083146] Lustre: Skipped 2 previous similar messages [ 8238.100613] Lustre: Skipped 1 previous similar message [ 8241.143701] Lustre: server umount lustre-OST0000 complete [ 8248.299669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.312680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8248.315586] Lustre: Skipped 1 previous similar message [ 8248.333263] Lustre: Skipped 1 previous similar message [ 8251.531224] Lustre: server umount lustre-MDT0000 complete [ 8253.410723] LustreError: 187104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.432988] LustreError: 187104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8255.574225] LustreError: 185872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053588 with bad export cookie 12121115984938758069 [ 8255.593238] LustreError: 185872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8255.602677] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8256.046777] Lustre: server umount lustre-MDT0001 complete [ 8265.838293] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:26:37 (1784053597) [ 8288.770776] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 8292.952926] Key type lgssc unregistered [ 8293.357644] LNet: 190794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8293.362724] LNetError: 190794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8293.374248] LNet: Removed LNI 192.168.201.129@tcp [ 8294.574841] Key type .llcrypt unregistered [ 8294.578065] Key type ._llcrypt unregistered [ 8316.388678] Key type ._llcrypt registered [ 8316.392183] Key type .llcrypt registered [ 8316.456308] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 8317.577815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8317.597526] alg: No test for adler32 (adler32-zlib) [ 8318.735268] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 8318.951608] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 8320.608621] Key type lgssc registered [ 8321.837771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8329.958746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8329.989749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8331.440634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8336.083556] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8341.992825] LustreError: 192013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8342.008532] LustreError: 192013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8344.864932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8345.331268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8349.838508] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8357.154801] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8362.863498] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8370.389554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8370.586831] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8370.589181] LustreError: 193875:0:(llog.c:1696:llog_backup()) MGC192.168.201.129@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8370.595560] Lustre: 193875:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.129@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8370.771618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.814865] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8385.876232] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8388.066814] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8388.075347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8388.076986] Lustre: Skipped 2 previous similar messages [ 8388.092803] Lustre: Skipped 2 previous similar messages [ 8390.113820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8390.121244] Lustre: Skipped 2 previous similar messages [ 8395.239141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8395.254539] Lustre: Skipped 2 previous similar messages [ 8400.352808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8400.362119] Lustre: Skipped 2 previous similar messages [ 8402.399173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8402.699596] Lustre: server umount lustre-MDT0000 complete [ 8405.473150] LustreError: 192014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8405.500237] LustreError: 192014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8406.163905] LustreError: 191993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053739 with bad export cookie 5987596492669228071 [ 8406.170695] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8406.175546] LustreError: 191993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8406.519379] LustreError: 192743:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000075071e6d x1870715841181184/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8406.634965] Lustre: server umount lustre-MDT0001 complete [ 8420.949662] Lustre: server umount lustre-OST0000 complete [ 8432.864474] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 8444.582860] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 8490.761435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.968106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8490.992679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8491.041309] Lustre: lustre-MDT0000: new disk, initializing [ 8491.089776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8491.104553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8496.165944] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8507.038425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.143348] Lustre: 197407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8507.196956] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8507.201068] Lustre: Skipped 1 previous similar message [ 8507.205650] LustreError: 198192:0:(llog.c:1696:llog_backup()) MGC192.168.201.129@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8507.215925] LustreError: 198192:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8507.228157] Lustre: 198192:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.129@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8507.262503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8507.265212] Lustre: Skipped 1 previous similar message [ 8507.349905] Lustre: lustre-MDT0001: new disk, initializing [ 8507.456300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8507.478884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8507.494226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8512.407625] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8524.556271] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8532.386813] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8542.478472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8542.870809] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8542.875884] Lustre: Skipped 1 previous similar message [ 8542.879567] LustreError: 199354:0:(llog.c:1696:llog_backup()) MGC192.168.201.129@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8542.891587] LustreError: 199354:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8542.902209] Lustre: 199354:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.129@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8542.912550] Lustre: 199354:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8542.953279] Lustre: lustre-OST0000: new disk, initializing [ 8542.960355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8542.984946] Lustre: 199378:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8543.143492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8550.102582] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8553.097294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8553.108285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8553.281913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8562.069541] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8567.614956] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8567.810915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8573.138567] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8573.378417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8579.054574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8579.057941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8579.087384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8579.112393] Lustre: Skipped 1 previous similar message [ 8581.562849] Lustre: server umount lustre-OST0000 complete [ 8589.280424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8589.285230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8589.290917] LustreError: Skipped 1 previous similar message [ 8589.292365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8589.317113] Lustre: Skipped 2 previous similar messages [ 8592.630780] Lustre: server umount lustre-MDT0000 complete [ 8594.402415] LustreError: 197415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8594.427989] LustreError: 197415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8596.621336] LustreError: 197399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053929 with bad export cookie 5987596492669228792 [ 8596.627692] LustreError: 197399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8596.632121] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8597.059471] Lustre: server umount lustre-MDT0001 complete [ 8608.349228] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:32:19 (1784053939) [ 8628.846420] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 8638.481353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8638.825582] LustreError: 202509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8638.868288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8642.630991] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8644.090307] LustreError: 202510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8651.934712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8657.590269] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8661.112429] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8668.716313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8676.474173] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8686.326918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8686.601314] Lustre: lustre-OST0001: new disk, initializing [ 8686.605603] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8686.611639] Lustre: 204819:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8686.705647] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8686.714539] Lustre: Skipped 2 previous similar messages [ 8693.513518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8693.538946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8693.839934] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8695.377292] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8708.679263] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8719.563064] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8721.439402] Lustre: DEBUG MARKER: ADD OST3 [ 8728.526837] loop: module loaded [ 8728.621963] loop0: detected capacity change from 0 to 204800000 [ 8728.991371] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8729.022445] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8729.035075] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8736.682422] loop0: detected capacity change from 0 to 204800000 [ 8737.086758] Lustre: DEBUG MARKER: STOP OST3 [ 8742.933608] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8743.138546] Lustre: lustre-OST0003: new disk, initializing [ 8743.143303] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8743.149744] Lustre: 206860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8744.853077] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8744.985071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8744.991028] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8745.031511] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8746.557826] Lustre: Failing over lustre-OST0003 [ 8746.686240] Lustre: server umount lustre-OST0003 complete [ 8746.912212] LustreError: 206986:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8750.067435] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8750.069416] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8750.088854] LustreError: 203996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8750.088864] LustreError: 203996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8750.276772] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8758.090210] Lustre: DEBUG MARKER: START OST3 again [ 8764.784699] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8765.124087] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8765.127886] Lustre: Skipped 1 previous similar message [ 8765.142504] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8765.583166] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8766.608444] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8766.609789] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8771.752495] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8793.757799] Lustre: Failing over lustre-OST0003 [ 8793.859765] Lustre: server umount lustre-OST0003 complete [ 8796.147153] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8796.159546] Lustre: Skipped 1 previous similar message [ 8796.165482] LustreError: 205113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8796.178174] LustreError: 205113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8816.108444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8816.123158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8816.146242] Lustre: Skipped 1 previous similar message [ 8816.159637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8816.166455] Lustre: Skipped 1 previous similar message [ 8820.625525] Lustre: server umount lustre-MDT0000 complete [ 8824.383463] LustreError: 202490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054157 with bad export cookie 5987596492669229674 [ 8824.393501] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8824.397564] LustreError: 202490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8829.858183] LustreError: 205544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8829.879527] LustreError: 205544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 8831.079718] Lustre: server umount lustre-MDT0001 complete [ 8835.559181] Lustre: server umount lustre-OST0000 complete [ 8839.392433] Lustre: server umount lustre-OST0001 complete [ 8847.434346] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 8857.480765] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 8900.182888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8900.696847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8900.737118] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8900.836141] Lustre: lustre-MDT0000: new disk, initializing [ 8900.954161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8900.980171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8906.882432] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8918.670641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.877981] Lustre: 212280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8918.963424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8918.966652] Lustre: Skipped 1 previous similar message [ 8919.074351] Lustre: lustre-MDT0001: new disk, initializing [ 8919.289862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8919.299596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8924.252204] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8935.074458] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8941.770524] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8949.552740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8949.861209] Lustre: 214252:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8951.301748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8951.313320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8951.461869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8958.264370] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 8969.086758] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8974.593376] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8974.825457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8979.668345] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8979.933792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8981.984081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8981.999450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8982.021495] Lustre: Skipped 5 previous similar messages [ 8982.025383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8982.028763] Lustre: Skipped 7 previous similar messages [ 8996.320754] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8996.427443] Lustre: server umount lustre-OST0000 complete [ 9001.440075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9001.456275] LustreError: Skipped 1 previous similar message [ 9014.241567] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9014.700230] Lustre: server umount lustre-MDT0000 complete [ 9018.351613] LustreError: 213160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9018.396643] LustreError: 213160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9019.664891] LustreError: 213087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054352 with bad export cookie 5987596492669231970 [ 9019.666894] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9019.673346] LustreError: 213087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9020.188953] Lustre: server umount lustre-MDT0001 complete [ 9025.951559] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:39:16 (1784054356) [ 9045.836384] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 9058.639610] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 9097.879618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.166060] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9098.245504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9098.253198] Lustre: Skipped 1 previous similar message [ 9098.330849] Lustre: lustre-MDT0000: new disk, initializing [ 9098.334812] Lustre: Skipped 1 previous similar message [ 9098.457557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9103.646472] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9116.250479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9116.405582] Lustre: 218868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9116.694131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9122.684412] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9134.139238] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9140.119858] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9149.600042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9150.242178] Lustre: 220837:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9151.419992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9151.553200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9157.078901] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9167.843956] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9173.765553] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9173.978650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9179.517148] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9179.771850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9187.300694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9187.310452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9187.330170] Lustre: Skipped 3 previous similar messages [ 9187.342943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9187.352834] Lustre: Skipped 13 previous similar messages [ 9188.880536] Lustre: server umount lustre-OST0000 complete [ 9208.287141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9208.429196] Lustre: server umount lustre-MDT0000 complete [ 9212.641622] LustreError: 218859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054545 with bad export cookie 5987596492669232852 [ 9212.646926] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9212.649094] LustreError: 218859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9212.916886] Lustre: server umount lustre-MDT0001 complete [ 9221.645058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9222.119182] LustreError: 223056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9222.203909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9222.206700] Lustre: Skipped 5 previous similar messages [ 9226.715790] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9235.766264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9242.744205] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9252.933185] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9260.720784] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9269.234728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9277.447604] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9286.236625] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9305.065089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 9305.076117] LustreError: Skipped 1 previous similar message [ 9305.086443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9305.112995] Lustre: Skipped 3 previous similar messages [ 9309.105361] Lustre: server umount lustre-OST0000 complete [ 9317.858458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9317.871279] Lustre: Skipped 11 previous similar messages [ 9319.623959] Lustre: server umount lustre-MDT0000 complete [ 9323.947676] LustreError: 223792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054656 with bad export cookie 5987596492669233727 [ 9323.956240] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9323.972920] LustreError: 223792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9324.782061] Lustre: server umount lustre-MDT0001 complete [ 9425.291645] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 9427.921797] Key type lgssc unregistered [ 9428.310142] LNet: 226740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9428.317824] LNetError: 226740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9429.349924] LNet: Removed LNI 192.168.201.129@tcp [ 9430.249201] Key type .llcrypt unregistered [ 9430.250818] Key type ._llcrypt unregistered [ 9456.692199] Key type ._llcrypt registered [ 9456.694342] Key type .llcrypt registered [ 9456.779533] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 9473.264895] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 9475.071666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9475.107805] alg: No test for adler32 (adler32-zlib) [ 9476.023255] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 9476.208649] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 9477.847217] Key type lgssc registered [ 9479.223376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9524.936739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9524.998774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9526.295703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9526.325524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9526.462578] Lustre: lustre-MDT0000: new disk, initializing [ 9526.576716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9526.597333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9531.826586] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9542.789920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9542.936830] Lustre: 230096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9542.993189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9542.998735] Lustre: Skipped 1 previous similar message [ 9543.070450] Lustre: lustre-MDT0001: new disk, initializing [ 9543.168835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9543.211689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9543.228186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9548.328255] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9557.969270] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9563.144336] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9569.621944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9570.030789] Lustre: lustre-OST0000: new disk, initializing [ 9570.034622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9570.048642] Lustre: 232066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9570.216696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9576.611947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9576.634838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9576.866905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9577.593871] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9587.376724] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9592.159058] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9592.371438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9597.489960] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9597.679615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9602.529256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9602.540953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9602.542941] LustreError: Skipped 1 previous similar message [ 9602.563249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9602.578669] Lustre: Skipped 1 previous similar message [ 9607.657306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9607.672271] Lustre: Skipped 2 previous similar messages [ 9612.778297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9612.791561] Lustre: Skipped 1 previous similar message [ 9614.303806] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9614.644787] Lustre: server umount lustre-OST0000 complete [ 9623.032987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9623.071586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9628.130379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9628.133274] Lustre: Skipped 1 previous similar message [ 9635.295247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9635.536096] Lustre: server umount lustre-MDT0000 complete [ 9640.335223] LustreError: 230089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054973 with bad export cookie 15854305703261080912 [ 9640.352059] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9640.366873] LustreError: 230089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9640.981996] Lustre: server umount lustre-MDT0001 complete [ 9665.982191] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 9680.532440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9681.247427] LustreError: 235003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9681.271260] LustreError: 235003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9681.300875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9686.500207] LustreError: 235004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9687.516798] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9691.624965] LustreError: 235003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9696.736850] LustreError: 235004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9696.844061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9697.223054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9701.776571] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9704.575332] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9711.330741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9711.809605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9717.703209] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9728.509429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9728.756490] Lustre: lustre-OST0001: new disk, initializing [ 9728.761493] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9728.768431] Lustre: 237313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9728.914415] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9734.741138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9734.758486] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9734.897773] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9735.733904] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9749.420843] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9757.522698] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9772.747786] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:51:44 (1784055104) [ 9776.101894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9776.113182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9776.121764] Lustre: Skipped 3 previous similar messages [ 9776.145968] Lustre: Skipped 5 previous similar messages [ 9789.408752] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9789.679429] Lustre: server umount lustre-MDT0000 complete [ 9791.457869] LustreError: 235004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9791.486332] LustreError: 235004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9793.787639] LustreError: 234983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055126 with bad export cookie 15854305703261081794 [ 9793.805464] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9794.266963] Lustre: server umount lustre-MDT0001 complete [ 9808.725752] Lustre: server umount lustre-OST0000 complete [ 9822.917839] Lustre: server umount lustre-OST0001 complete [ 9829.375471] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 9844.917173] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 9892.354052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9892.884942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9892.926473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9893.091168] Lustre: lustre-MDT0000: new disk, initializing [ 9893.262100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9893.311054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9899.511227] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9912.290357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9912.471175] Lustre: 241285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9912.610896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9912.615556] Lustre: Skipped 1 previous similar message [ 9912.795942] Lustre: lustre-MDT0001: new disk, initializing [ 9913.034482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9913.049745] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9920.271299] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9935.257847] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9943.065554] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9951.234281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9951.788365] Lustre: lustre-OST0000: new disk, initializing [ 9951.804460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9951.813926] Lustre: 243256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9953.517376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9953.537739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9953.716141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9959.525511] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 9970.967634] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9977.214611] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9977.535408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9982.316395] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9982.751815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9989.608471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9989.615151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9989.635884] Lustre: Skipped 1 previous similar message [ 9989.654859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9989.660053] Lustre: Skipped 9 previous similar messages [ 9991.821954] Lustre: server umount lustre-OST0000 complete [ 9999.840672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9999.842160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9999.844148] LustreError: Skipped 1 previous similar message [ 9999.869924] Lustre: Skipped 2 previous similar messages [10002.421570] Lustre: server umount lustre-MDT0000 complete [10004.961182] LustreError: 241297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10004.988181] LustreError: 241297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10006.571278] LustreError: 241278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055339 with bad export cookie 15854305703261083929 [10006.576588] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10006.580816] LustreError: 241278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10007.216063] Lustre: server umount lustre-MDT0001 complete [10016.788893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10017.643581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10017.655592] Lustre: Skipped 2 previous similar messages [10022.889358] LustreError: 245473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10022.906560] LustreError: 245473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10023.280942] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10033.428845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10038.748067] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10046.509939] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10052.625899] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10059.885607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10067.001446] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10075.417146] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10227.168972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10227.191207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10227.219362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10227.222253] Lustre: Skipped 2 previous similar messages [10229.674406] Lustre: server umount lustre-OST0000 complete [10241.004676] Lustre: server umount lustre-MDT0000 complete [10242.533878] LustreError: 245473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10242.566856] LustreError: 245473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10245.523465] LustreError: 245455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055578 with bad export cookie 15854305703261084811 [10245.532482] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10245.533539] LustreError: 245455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10246.157440] Lustre: server umount lustre-MDT0001 complete [10348.783865] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [10352.268630] Key type lgssc unregistered [10352.513746] LNet: 249093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10352.519827] LNetError: 249093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10352.533856] LNet: Removed LNI 192.168.201.129@tcp [10353.286153] Key type .llcrypt unregistered [10353.290209] Key type ._llcrypt unregistered [10382.231676] Key type ._llcrypt registered [10382.236996] Key type .llcrypt registered [10382.393702] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [10399.627810] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [10400.536165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10400.644162] alg: No test for adler32 (adler32-zlib) [10401.711186] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [10402.014696] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [10403.727192] Key type lgssc registered [10405.133733] Lustre: Echo OBD driver; http://www.lustre.org/ [10465.964886] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [10480.527032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10480.569276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10482.026400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10482.066726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10482.208457] Lustre: lustre-MDT0000: new disk, initializing [10482.321873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10482.346857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10487.039955] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10499.642398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10499.750940] Lustre: 253186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10499.800641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10499.804355] Lustre: Skipped 1 previous similar message [10499.866704] Lustre: lustre-MDT0001: new disk, initializing [10499.950293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10499.994783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10500.011275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10505.512316] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10512.257774] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10521.464361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10521.927295] Lustre: lustre-OST0000: new disk, initializing [10521.930866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10521.960158] Lustre: 254818:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10522.117768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10525.738097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10525.765098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10525.937412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10530.287460] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10542.993830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10543.177787] Lustre: lustre-OST0001: new disk, initializing [10543.180741] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10543.185899] Lustre: 255690:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10543.308691] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10550.247586] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10551.952364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10551.963631] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10552.174961] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10562.151240] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10569.289538] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10581.984631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10581.990855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10582.007127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10583.012385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10583.015786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10583.037548] Lustre: Skipped 1 previous similar message [10587.982693] Lustre: server umount lustre-MDT0000 complete [10591.506898] LustreError: 253179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055924 with bad export cookie 12201399537835323665 [10591.513766] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10591.522336] LustreError: 253179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10591.774864] Lustre: server umount lustre-MDT0001 complete [10606.983553] Lustre: server umount lustre-OST0000 complete [10609.695718] Lustre: 250671:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784055926/real 1784055926] req@ffff8d074850fb80 x1870718026463232/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784055942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10609.734113] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10609.754722] Lustre: Skipped 1 previous similar message [10611.616371] Lustre: server umount lustre-OST0001 complete [10626.455225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10638.900379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10650.839756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10663.898684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10686.735760] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [10699.716357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10699.749406] Lustre: lustre-MDT0000: reset Object Index mappings [10700.276545] LustreError: 260702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.297963] LustreError: 260702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10700.447792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10705.877443] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10705.890780] LustreError: 260703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10710.005434] LustreError: 260702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.112520] LustreError: 260703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.145817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10715.161757] Lustre: lustre-MDT0001: reset Object Index mappings [10715.493732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10720.355366] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10724.052050] Lustre: 261845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10731.003033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10731.580739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10738.742554] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10740.836889] LustreError: 262199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10746.025651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10749.190093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10755.095479] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10756.727458] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10765.774769] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10769.798209] Lustre: 263716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10775.524742] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10775.531860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10775.550501] Lustre: Skipped 2 previous similar messages [10775.580745] Lustre: Skipped 6 previous similar messages [10776.733400] Lustre: server umount lustre-MDT0000 complete [10780.649575] LustreError: 260697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10780.677359] LustreError: 260697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10780.867193] LustreError: 263027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056113 with bad export cookie 12201399537835330455 [10780.869724] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10780.877624] LustreError: 263027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10781.238857] Lustre: server umount lustre-MDT0001 complete [10795.137154] Lustre: server umount lustre-OST0000 complete [10809.992911] Lustre: server umount lustre-OST0001 complete [10840.318277] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:09:31 (1784056171) [10860.188694] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [10876.632905] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [10920.566774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10920.794831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10920.822395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10920.893897] Lustre: lustre-MDT0000: new disk, initializing [10920.981759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10920.987872] Lustre: Skipped 1 previous similar message [10921.016863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10926.418880] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10938.870730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10939.100818] Lustre: 267995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10939.204952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10939.207892] Lustre: Skipped 1 previous similar message [10939.319804] Lustre: lustre-MDT0001: new disk, initializing [10939.504328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10939.515044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10944.951645] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10956.547512] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10964.263773] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10972.027755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10972.354962] Lustre: lustre-OST0000: new disk, initializing [10972.358955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10972.368862] Lustre: 269967:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10974.057427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10974.071453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10974.188476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10978.952516] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [10989.420740] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10995.481067] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10995.860524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11001.291619] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11001.516238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11004.903312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11004.923775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11004.933790] LustreError: Skipped 1 previous similar message [11004.934690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11004.992305] Lustre: Skipped 2 previous similar messages [11010.053142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11010.082821] Lustre: Skipped 2 previous similar messages [11018.211643] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11018.577556] Lustre: server umount lustre-OST0000 complete [11025.379475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11025.388544] Lustre: Skipped 1 previous similar message [11025.389212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11025.406332] Lustre: Skipped 3 previous similar messages [11038.175163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11038.675496] Lustre: server umount lustre-MDT0000 complete [11040.741031] LustreError: 268846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11040.741031] LustreError: 268003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11043.400129] LustreError: 267989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056376 with bad export cookie 12201399537835332548 [11043.404380] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11043.414393] LustreError: 267989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11044.069325] Lustre: server umount lustre-MDT0001 complete [11066.954604] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [11078.487755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11078.856593] LustreError: 272904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11078.921706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11078.928712] Lustre: Skipped 2 previous similar messages [11084.401511] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11094.569406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11100.491915] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11103.876502] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11111.907108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11119.945755] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11130.800190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11131.011649] Lustre: lustre-OST0001: new disk, initializing [11131.017987] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11131.028598] Lustre: 275213:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11132.481347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11132.490659] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11132.623220] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11137.940054] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11148.395741] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11153.359283] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11159.812351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11159.878568] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11168.742773] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11168.745749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11168.758138] Lustre: Skipped 2 previous similar messages [11168.766920] Lustre: Skipped 7 previous similar messages [11173.607393] Lustre: server umount lustre-MDT0000 complete [11173.870504] LustreError: 274052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11173.887464] LustreError: 274052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [11177.807569] LustreError: 272886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056510 with bad export cookie 12201399537835333430 [11177.816038] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11178.084754] Lustre: server umount lustre-MDT0001 complete [11192.848340] Lustre: server umount lustre-OST0000 complete [11195.362461] Lustre: 250670:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784056512/real 1784056512] req@ffff8d0753058e00 x1870718026633088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784056528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11196.867479] Lustre: server umount lustre-OST0001 complete [11218.277953] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [11229.796658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11230.213864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11230.222294] Lustre: Skipped 3 previous similar messages [11235.805595] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11245.348519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11250.392608] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11253.378570] Lustre: 279412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11260.286600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11268.280246] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11275.189189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11277.722037] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11283.440499] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11284.743646] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11294.011266] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11298.638053] Lustre: 281289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11304.530678] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11304.601878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11315.766443] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:17:27 (1784056647) [11333.088779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11333.092326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11333.102758] Lustre: Skipped 2 previous similar messages [11333.106651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11337.048744] Lustre: server umount lustre-MDT0000 complete [11339.753375] LustreError: 279293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11339.792311] LustreError: 279293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11342.560351] LustreError: 278252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056675 with bad export cookie 12201399537835340220 [11342.583863] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11349.448841] Lustre: server umount lustre-MDT0001 complete [11362.441523] Lustre: server umount lustre-OST0000 complete [11367.276406] Lustre: server umount lustre-OST0001 complete [11377.228898] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [11389.809247] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [11443.916813] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [11455.066303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11455.365831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11455.408262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11455.520902] Lustre: lustre-MDT0000: new disk, initializing [11455.650516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11460.528243] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11472.618590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11472.746336] Lustre: 285530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11472.958055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11478.124894] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11489.472947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11489.760604] Lustre: 287164:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11489.834576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11489.847537] Lustre: Skipped 5 previous similar messages [11491.922437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11496.835376] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11507.807566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11507.948217] Lustre: 288035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11509.759477] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11509.777073] Lustre: Skipped 1 previous similar message [11509.887595] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11516.129879] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11526.810536] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11530.705978] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11530.709310] Lustre: Skipped 1 previous similar message [11548.563881] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:21:19 (1784056879) [11550.695988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11550.700826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11550.717505] Lustre: Skipped 6 previous similar messages [11550.727390] Lustre: Skipped 8 previous similar messages [11553.047506] Lustre: server umount lustre-MDT0000 complete [11557.127067] LustreError: 287174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056890 with bad export cookie 12201399537835347633 [11557.131750] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11557.135740] LustreError: 287174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11557.635147] Lustre: server umount lustre-MDT0001 complete [11571.679322] Lustre: server umount lustre-OST0000 complete [11585.928330] Lustre: server umount lustre-OST0001 complete [11596.188376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11596.638701] LustreError: 290284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11596.659610] LustreError: 290284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [11602.709992] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11614.823508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11621.033343] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11629.524589] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11636.393981] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11643.943884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11651.569296] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11660.139917] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11688.192381] loop1: detected capacity change from 0 to 204800000 [11688.484871] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11688.495768] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11688.503942] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11693.597152] loop1: detected capacity change from 0 to 204800000 [11699.598918] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11699.668936] LustreError: 290274:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs10726: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11699.689573] LustreError: 290274:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs10726 [11699.697239] LustreError: 290274:0:(mgs_handler.c:517:mgs_target_reg()) fs10726-MDT0001: Error checking target index: rc = -13 [11699.711438] LustreError: fs10726-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11699.720903] LustreError: 294023:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11699.730367] LustreError: 294023:0:(tgt_mount.c:2129:server_put_super()) no obd fs10726-MDT0001 [11699.734269] LustreError: 294023:0:(tgt_mount.c:116:server_deregister_mount()) fs10726-MDT0001 not registered [11699.777264] Lustre: server umount fs10726-MDT0001 complete [11699.780414] LustreError: 294023:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11708.443308] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11708.565756] Lustre: Setting parameter fs10726-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs10726-MDT0001 [11708.610633] Lustre: srv-fs10726-MDT0001: No data found on store. Initialize space. [11708.616025] Lustre: Skipped 4 previous similar messages [11708.750758] Lustre: fs10726-MDT0001: new disk, initializing [11708.753612] Lustre: Skipped 3 previous similar messages [11708.922786] Lustre: srv-fs10726-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11708.928544] Lustre: 294489:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10726-MDT0001-mdtlov: can't create new llog: rc = -115 [11708.932804] LustreError: 294489:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10726-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11713.617913] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11720.840895] Lustre: server umount fs10726-MDT0001 complete [11733.161822] loop1: detected capacity change from 0 to 204800000 [11738.595470] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11738.902749] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11739.395482] Lustre: 296058:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10726-MDT0001-mdtlov: can't create new llog: rc = -115 [11739.409390] LustreError: 296058:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10726-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11743.736315] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11744.889319] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11750.115481] Lustre: server umount fs10726-MDT0001 complete [11763.360030] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.29@tcp (stopping) [11763.366819] Lustre: Skipped 3 previous similar messages [11763.681173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11763.691083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11763.709708] Lustre: Skipped 2 previous similar messages [11768.029885] Lustre: server umount lustre-MDT0000 complete [11777.807326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11778.000399] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11782.779332] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11783.670473] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11790.123420] loop1: detected capacity change from 0 to 204800000 [11795.385281] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11795.743875] Lustre: 298504:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10726-MDT0001-mdtlov: can't create new llog: rc = -115 [11795.749046] LustreError: 298504:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10726-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11800.608176] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11805.612928] Lustre: server umount fs10726-MDT0001 complete [11829.739811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11829.749892] LustreError: Skipped 1 previous similar message [11835.112802] Lustre: server umount lustre-OST0000 complete [11845.683876] Lustre: server umount lustre-MDT0000 complete [11849.283255] LustreError: 290265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057182 with bad export cookie 12201399537835350314 [11849.290429] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11849.771373] Lustre: server umount lustre-MDT0001 complete [11859.811684] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [11862.130994] Key type lgssc unregistered [11862.350180] LNet: 300852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11862.355234] LNetError: 300852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11862.366599] LNet: Removed LNI 192.168.201.129@tcp [11863.128158] Key type .llcrypt unregistered [11863.129927] Key type ._llcrypt unregistered [11884.445306] Key type ._llcrypt registered [11884.451315] Key type .llcrypt registered [11884.613702] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [11886.385703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11886.441614] alg: No test for adler32 (adler32-zlib) [11887.557397] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [11887.904068] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [11889.623180] Key type lgssc registered [11890.915769] Lustre: Echo OBD driver; http://www.lustre.org/ [11898.040712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11898.063576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11899.573768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11903.926191] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11909.108119] LustreError: 302123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.129912] LustreError: 302123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11913.271538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11913.768308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11917.501948] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11924.818894] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11930.801966] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11937.518948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11937.820170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11938.861839] LustreError: 303951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11943.909795] LustreError: 303950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11944.002927] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11950.055915] LustreError: 303950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11950.081319] LustreError: 303950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11951.693755] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11955.177115] LustreError: 303953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11955.210217] LustreError: 303953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11964.056772] LustreError: 303951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11964.090374] LustreError: 303951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11968.929565] loop1: detected capacity change from 0 to 204800000 [11969.183245] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11969.205071] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11969.216466] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11974.379040] loop1: detected capacity change from 0 to 204800000 [11979.968699] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11980.108634] Lustre: fs10444-OST0000: new disk, initializing [11980.111929] Lustre: srv-fs10444-OST0000: No data found on store. Initialize space. [11980.120129] Lustre: 305858:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11980.222809] Lustre: fs10444-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11980.772192] LustreError: 303953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.802669] LustreError: 303953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11986.735319] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [11995.777799] Lustre: server umount fs10444-OST0000 complete [12008.792556] loop1: detected capacity change from 0 to 204800000 [12014.444884] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12014.705284] Lustre: fs10444-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12015.245839] LustreError: 303950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12015.267903] LustreError: 303950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [12020.418724] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12021.744791] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12026.861593] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12028.973702] loop2: detected capacity change from 0 to 204800000 [12029.202528] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12029.213992] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12029.220546] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12034.123705] loop2: detected capacity change from 0 to 204800000 [12040.066956] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12040.177827] LustreError: fs10444-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12040.183135] LustreError: fs10444-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12040.188754] LustreError: 308897:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12040.193654] LustreError: 308897:0:(tgt_mount.c:2129:server_put_super()) no obd fs10444-OST0001 [12040.198151] LustreError: 308897:0:(tgt_mount.c:116:server_deregister_mount()) fs10444-OST0001 not registered [12040.283488] Lustre: server umount fs10444-OST0001 complete [12040.292418] LustreError: 308897:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [12051.966651] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12052.157362] Lustre: fs10444-OST0001: new disk, initializing [12052.167394] Lustre: srv-fs10444-OST0001: No data found on store. Initialize space. [12052.176283] Lustre: 309448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12052.309681] Lustre: fs10444-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12059.345817] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12068.868360] Lustre: server umount fs10444-OST0000 complete [12077.957382] Lustre: server umount fs10444-OST0001 complete [12081.807657] LustreError: 303951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12081.840922] LustreError: 303951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [12089.340450] loop1: detected capacity change from 0 to 204800000 [12094.864139] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12095.079508] Lustre: fs10444-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12100.977844] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12107.241384] loop2: detected capacity change from 0 to 204800000 [12112.188945] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12118.673442] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12127.206674] Lustre: server umount fs10444-OST0000 complete [12136.282989] Lustre: server umount fs10444-OST0001 complete [12149.736728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.747279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12149.748428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12149.758393] Lustre: Skipped 1 previous similar message [12149.777016] Lustre: Skipped 1 previous similar message [12154.849599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12154.857897] Lustre: Skipped 1 previous similar message [12155.293518] Lustre: server umount lustre-OST0000 complete [12159.461647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12159.467358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12159.479645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12165.090033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12165.100208] Lustre: Skipped 2 previous similar messages [12165.631799] Lustre: server umount lustre-MDT0000 complete [12169.288691] LustreError: 302104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057502 with bad export cookie 13237103139263684222 [12169.296087] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12169.297351] LustreError: 302104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12169.643215] Lustre: server umount lustre-MDT0001 complete [12180.294706] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [12182.953979] Key type lgssc unregistered [12183.215748] LNet: 315372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12183.224435] LNetError: 315372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12183.253823] LNet: Removed LNI 192.168.201.129@tcp [12184.054129] Key type .llcrypt unregistered [12184.063722] Key type ._llcrypt unregistered [12217.819834] Key type ._llcrypt registered [12217.824668] Key type .llcrypt registered [12217.929724] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [12235.022923] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [12236.122250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12236.246875] alg: No test for adler32 (adler32-zlib) [12237.240146] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [12237.497058] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [12239.175170] Key type lgssc registered [12240.402395] Lustre: Echo OBD driver; http://www.lustre.org/ [12277.089298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12277.104650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12278.302855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12278.334244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12278.426585] Lustre: lustre-MDT0000: new disk, initializing [12278.512561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12278.548375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12282.287461] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12291.459487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12291.576471] Lustre: 319235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12291.627726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12291.631826] Lustre: Skipped 1 previous similar message [12291.686126] Lustre: lustre-MDT0001: new disk, initializing [12291.767467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12291.799738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12291.808423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12295.870231] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12304.686285] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12310.123679] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12317.226160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12317.598847] Lustre: lustre-OST0000: new disk, initializing [12317.602866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12317.608611] Lustre: 321202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12317.702192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12323.918337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12323.925149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12324.119523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12324.451513] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12335.108982] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12337.967593] LustreError: 317408:0:(lmv_obd.c:208:lmv_notify()) activation of lustre-MDT0001_UUID failed: -22 [12338.013957] Lustre: Mounted lustre-client [12339.240886] Lustre: Unmounted lustre-client [12353.516215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12353.522965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12353.543447] Lustre: Skipped 1 previous similar message [12353.573863] Lustre: Skipped 1 previous similar message [12358.626073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12358.639579] Lustre: Skipped 1 previous similar message [12363.745055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12363.767961] Lustre: Skipped 1 previous similar message [12365.279332] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12365.478548] Lustre: server umount lustre-OST0000 complete [12373.985438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12373.995497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12374.005618] Lustre: Skipped 1 previous similar message [12374.023715] Lustre: Skipped 1 previous similar message [12375.924415] Lustre: server umount lustre-MDT0000 complete [12379.386838] LustreError: 319225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057712 with bad export cookie 10321568611884757564 [12379.388332] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12379.401934] LustreError: 319225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12380.016081] Lustre: server umount lustre-MDT0001 complete [12391.922558] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [12393.193419] LNet: 1 local NIs in recovery (showing 1): 192.168.201.129@tcp [12394.958901] Key type lgssc unregistered [12395.300282] LNet: 323365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12395.304981] LNetError: 323365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12470.055323] LNet: Removed LNI 192.168.201.129@tcp [12470.958653] Key type .llcrypt unregistered [12470.960468] Key type ._llcrypt unregistered [12490.801663] Key type ._llcrypt registered [12490.805606] Key type .llcrypt registered [12490.983464] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [12510.683558] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [12511.973959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12512.008683] alg: No test for adler32 (adler32-zlib) [12513.089937] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [12513.363334] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [12515.023173] Key type lgssc registered [12516.303608] Lustre: Echo OBD driver; http://www.lustre.org/ [12549.803185] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:38:01 (1784057881) [12556.619664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12556.640793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12557.888069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12557.919950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12557.998794] Lustre: lustre-MDT0000: new disk, initializing [12558.073986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12558.084689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12562.969355] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12573.364809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12573.452644] Lustre: 326592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12573.491808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12573.495081] Lustre: Skipped 1 previous similar message [12573.545944] Lustre: lustre-MDT0001: new disk, initializing [12573.614939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12573.649422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12573.665253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12577.995937] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12588.955624] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12595.530184] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12603.068750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12603.332474] Lustre: lustre-OST0000: new disk, initializing [12603.335869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12603.344826] Lustre: 328561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12603.440968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12610.454303] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12611.617713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12611.631620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12611.747274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12619.355561] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12637.152829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12637.166372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12637.203265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12637.664408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12637.674690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12637.711249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12640.760242] Lustre: server umount lustre-OST0000 complete [12647.925578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12647.940780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12647.964412] Lustre: Skipped 1 previous similar message [12648.022342] Lustre: Skipped 1 previous similar message [12653.024265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12657.122627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12657.130957] Lustre: Skipped 2 previous similar messages [12660.191277] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12660.488450] Lustre: server umount lustre-MDT0000 complete [12664.137033] LustreError: 326586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057997 with bad export cookie 12729922727742270026 [12664.155157] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12664.501959] Lustre: server umount lustre-MDT0001 complete [12675.834714] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [12678.658391] Key type lgssc unregistered [12678.976928] LNet: 330470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12678.983914] LNetError: 330470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12678.998863] LNet: Removed LNI 192.168.201.129@tcp [12679.869821] Key type .llcrypt unregistered [12679.871449] Key type ._llcrypt unregistered [12702.040097] Key type ._llcrypt registered [12702.043444] Key type .llcrypt registered [12702.164607] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [12703.588609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12703.628682] alg: No test for adler32 (adler32-zlib) [12704.731631] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [12705.146803] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [12706.817614] Key type lgssc registered [12708.020052] Lustre: Echo OBD driver; http://www.lustre.org/ [12715.279372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12715.299433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12716.587113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12720.711563] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12726.758084] LustreError: 331746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12726.770568] LustreError: 331746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12728.375216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12728.708829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12732.989328] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12741.578447] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12750.094377] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12758.936650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12759.465252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12766.278379] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12775.965319] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12786.734741] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12788.633864] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12794.336693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12794.346589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12794.349744] Lustre: Skipped 2 previous similar messages [12794.373241] Lustre: Skipped 2 previous similar messages [12798.435868] Lustre: server umount lustre-MDT0000 complete [12799.462500] LustreError: 331746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12799.492917] LustreError: 331746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12802.787659] LustreError: 333588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784058135 with bad export cookie 16312021526674974668 [12802.798581] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12802.801361] LustreError: 333588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12804.576918] LustreError: 331742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12804.577744] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12804.598039] LustreError: 331742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12804.614581] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12804.617515] Lustre: Skipped 1 previous similar message [12807.585044] LustreError: 333698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12807.585072] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12807.603338] LustreError: 333698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12807.645882] Lustre: Skipped 1 previous similar message [12809.891277] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.29@tcp (stopping) [12812.707014] LustreError: 334236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12812.719861] LustreError: 334236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12815.016415] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.29@tcp (stopping) [12815.027419] Lustre: Skipped 1 previous similar message [12817.375145] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12817.648631] Lustre: server umount lustre-MDT0001 complete [12822.697165] Lustre: server umount lustre-OST0000 complete [12833.128580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12838.730805] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12841.096202] Lustre: 335931:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.129@tcp [12843.225766] Lustre: server umount MGS complete [12851.505986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12851.876500] LustreError: 336504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12851.897794] LustreError: 336504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12851.938660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12856.274426] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12865.950083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12866.371691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12871.148785] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12880.722964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12881.017699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12887.564905] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [12895.893617] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12897.708289] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12903.718231] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12905.434084] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12912.104468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12912.104926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12912.109253] LustreError: Skipped 1 previous similar message [12912.125606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12912.134934] Lustre: Skipped 1 previous similar message [12917.051815] Lustre: server umount lustre-OST0000 complete [12922.343836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12922.367740] Lustre: Skipped 2 previous similar messages [12927.090635] Lustre: server umount lustre-MDT0000 complete [12927.461812] LustreError: 336505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12927.481576] LustreError: 336505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12930.426305] LustreError: 336485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784058263 with bad export cookie 16312021526674975501 [12930.430259] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12930.446330] LustreError: 336485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12930.867093] Lustre: server umount lustre-MDT0001 complete [12941.308278] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [12944.148493] Key type lgssc unregistered [12944.406073] LNet: 339941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12944.412858] LNetError: 339941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12944.434371] LNet: Removed LNI 192.168.201.129@tcp [12945.292072] Key type .llcrypt unregistered [12945.303251] Key type ._llcrypt unregistered [12964.697389] Key type ._llcrypt registered [12964.699346] Key type .llcrypt registered [12964.780529] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [12982.360477] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [12983.578873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12983.628850] alg: No test for adler32 (adler32-zlib) [12984.676927] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [12984.956954] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [12986.615153] Key type lgssc registered [12987.724772] Lustre: Echo OBD driver; http://www.lustre.org/ [13024.228475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13024.271518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13025.579645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13025.617895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13025.719917] Lustre: lustre-MDT0000: new disk, initializing [13025.803760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13025.824353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13030.147973] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [13040.196169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13040.344304] Lustre: 343103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13040.423647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13040.428356] Lustre: Skipped 1 previous similar message [13040.500305] Lustre: lustre-MDT0001: new disk, initializing [13040.597500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13040.628777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13040.648666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13045.173541] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [13054.853932] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13060.691917] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13066.896907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13067.134655] Lustre: lustre-OST0000: new disk, initializing [13067.140878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13067.146450] Lustre: 345069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13067.264426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13072.985518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13072.997047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13073.123829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13073.241230] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [13082.907857] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13087.466286] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13087.645938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13091.947349] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13092.307114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13098.978312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13098.980409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13098.990700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13099.005039] Lustre: Skipped 1 previous similar message [13099.020049] Lustre: Skipped 1 previous similar message [13100.611445] Lustre: server umount lustre-OST0000 complete [13107.169102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13107.174633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13107.190831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13109.225226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13109.237202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13110.540583] Lustre: server umount lustre-MDT0000 complete [13114.359908] LustreError: 345067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784058447 with bad export cookie 12752261459689036453 [13114.360731] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13114.376080] LustreError: 345067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13114.727257] Lustre: server umount lustre-MDT0001 complete [13124.776645] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 15:47:36 (1784058456) [13142.891638] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [13152.908796] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [13188.599439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13188.810597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13188.831515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13188.922361] Lustre: lustre-MDT0000: new disk, initializing [13189.030974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13189.049362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13193.508348] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [13204.398537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13204.598814] Lustre: 349834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13204.677236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13204.681022] Lustre: Skipped 1 previous similar message [13204.772634] Lustre: lustre-MDT0001: new disk, initializing [13204.938726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13204.983170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13204.996486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13210.500427] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [13222.258644] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13230.150841] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13239.484587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13239.990786] Lustre: lustre-OST0000: new disk, initializing [13240.003518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13240.013259] Lustre: 351800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13240.164477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13247.622659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13247.645935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13247.820749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13248.373732] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [13258.319685] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13299.174155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13299.175091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13299.192531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13299.192547] Lustre: Skipped 1 previous similar message [13299.197566] LustreError: Skipped 1 previous similar message [13300.940845] Lustre: server umount lustre-OST0000 complete [13307.362712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13307.370850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13307.384207] Lustre: Skipped 1 previous similar message [13307.390058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13312.752338] Lustre: server umount lustre-MDT0000 complete [13314.537806] LustreError: 349838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13314.563918] LustreError: 349838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13317.981600] LustreError: 349825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784058651 with bad export cookie 12752261459689037335 [13317.995086] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13317.996172] LustreError: 349825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13318.016864] LustreError: 341325:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [13318.557396] Lustre: server umount lustre-MDT0001 complete [13333.798951] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13336.910920] Key type lgssc unregistered [13337.298350] LNet: 353973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13337.304442] LNetError: 353973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13337.329986] LNet: Removed LNI 192.168.201.129@tcp [13338.414144] Key type .llcrypt unregistered [13338.419979] Key type ._llcrypt unregistered [13445.325536] Key type ._llcrypt registered [13445.328795] Key type .llcrypt registered [13445.413933] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13445.669517] Key type .llcrypt unregistered [13445.673682] Key type ._llcrypt unregistered [13526.592156] Key type ._llcrypt registered [13526.594431] Key type .llcrypt registered [13526.675905] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13526.893588] Key type .llcrypt unregistered [13526.895766] Key type ._llcrypt unregistered [13560.881145] Key type ._llcrypt registered [13560.882428] Key type .llcrypt registered [13561.052901] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13561.451255] Key type .llcrypt unregistered [13561.457143] Key type ._llcrypt unregistered [13592.775316] Key type ._llcrypt registered [13592.778783] Key type .llcrypt registered [13592.916114] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13593.188791] Key type .llcrypt unregistered [13593.193846] Key type ._llcrypt unregistered [13685.050653] Key type ._llcrypt registered [13685.052558] Key type .llcrypt registered [13685.166870] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13685.450372] Key type .llcrypt unregistered [13685.452378] Key type ._llcrypt unregistered [13722.645302] Key type ._llcrypt registered [13722.647694] Key type .llcrypt registered [13722.767570] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13723.144545] Key type .llcrypt unregistered [13723.146808] Key type ._llcrypt unregistered [13759.678873] Key type ._llcrypt registered [13759.680798] Key type .llcrypt registered [13759.833439] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13760.157310] Key type .llcrypt unregistered [13760.159205] Key type ._llcrypt unregistered [13792.767076] Key type ._llcrypt registered [13792.769658] Key type .llcrypt registered [13792.907512] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13793.343377] Key type .llcrypt unregistered [13793.346438] Key type ._llcrypt unregistered [13829.615691] Key type ._llcrypt registered [13829.617577] Key type .llcrypt registered [13829.699371] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13829.975473] Key type .llcrypt unregistered [13829.980298] Key type ._llcrypt unregistered [13867.956799] Key type ._llcrypt registered [13867.961176] Key type .llcrypt registered [13868.046657] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13868.302488] Key type .llcrypt unregistered [13868.306509] Key type ._llcrypt unregistered [13942.289115] Key type ._llcrypt registered [13942.292660] Key type .llcrypt registered [13942.488471] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13943.017342] Key type .llcrypt unregistered [13943.019315] Key type ._llcrypt unregistered [13977.121970] Key type ._llcrypt registered [13977.124495] Key type .llcrypt registered [13977.217937] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [13977.449456] Key type .llcrypt unregistered [13977.451819] Key type ._llcrypt unregistered [14012.306185] Key type ._llcrypt registered [14012.310603] Key type .llcrypt registered [14012.462601] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [14012.865201] Key type .llcrypt unregistered [14012.878635] Key type ._llcrypt unregistered [14070.635479] Key type ._llcrypt registered [14070.639642] Key type .llcrypt registered [14070.758923] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [14090.328471] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [14091.685677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14091.696615] alg: No test for adler32 (adler32-zlib) [14092.862568] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [14093.241679] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [14094.968543] Key type lgssc registered [14096.902686] Lustre: Echo OBD driver; http://www.lustre.org/ [14138.189204] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14138.233626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14139.709782] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14139.782281] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14139.936208] Lustre: tst250fs-MDT0000: new disk, initializing [14140.106590] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14140.124856] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14144.778474] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [14157.973055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14158.160740] Lustre: 374350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14158.281898] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14158.288063] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (not set up) [14158.289427] Lustre: Skipped 1 previous similar message [14158.366826] Lustre: tst250fs-MDT0001: new disk, initializing [14158.522965] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14158.622625] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14158.642700] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14164.195974] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [14169.596464] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14176.489154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14176.705449] Lustre: tst250fs-OST0000: new disk, initializing [14176.709334] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14176.714954] Lustre: 375984:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14176.811368] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14179.075231] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14179.087559] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14179.190423] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14183.744950] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [14195.690139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14195.876824] Lustre: tst250fs-OST0001: new disk, initializing [14195.882324] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14195.890521] Lustre: 376855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14195.981457] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14203.850593] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [14206.009052] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14206.025573] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14206.088799] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14209.874677] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14211.949187] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14211.952970] Lustre: Skipped 1 previous similar message [14235.616265] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14235.630394] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14235.651509] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14237.155807] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14237.166621] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14240.958536] Lustre: server umount tst250fs-MDT0000 complete [14245.875926] LustreError: 375147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784059578 with bad export cookie 3696419547830094878 [14245.879189] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14245.881421] LustreError: 375147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14246.321453] Lustre: server umount tst250fs-MDT0001 complete [14261.490662] Lustre: server umount tst250fs-OST0000 complete [14263.391494] Lustre: 372577:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784059580/real 1784059580] req@ffff8d074b651500 x1870721896752896/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784059596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14263.408282] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14263.426358] Lustre: Skipped 2 previous similar messages [14266.999353] Lustre: server umount tst250fs-OST0001 complete [14289.202468] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [14299.444396] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [14338.624611] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:07:49 (1784059669) [14341.127591] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14361.518026] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [14375.752576] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [14413.138921] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14112 sec ========= 16:09:04 (1784059744) [14415.400118] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:09:06 (1784059746) === [14419.476656] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:09:11 (1784059751) === [14469.482073] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [14472.436865] Key type lgssc unregistered [14472.852478] LNet: 386021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14472.863746] LNetError: 386021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14472.896118] LNet: Removed LNI 192.168.201.129@tcp [14474.110170] Key type .llcrypt unregistered [14474.112405] Key type ._llcrypt unregistered