[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474403282 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002381] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006862] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009177] LSM: Security Framework initializing [ 0.010087] Yama: becoming mindful. [ 0.011045] SELinux: Initializing. [ 0.013026] *** VALIDATE selinux *** [ 0.021652] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026114] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] *** VALIDATE tmpfs *** [ 0.030483] *** VALIDATE proc *** [ 0.032293] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035005] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042410] debug: unmapping init [mem 0xffffffffa2a59000-0xffffffffa2a60fff] [ 0.044177] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045706] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046035] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048017] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053284] rcu: Hierarchical SRCU implementation. [ 0.055384] smp: Bringing up secondary CPUs ... [ 0.056825] x86: Booting SMP configuration: [ 0.057035] .... node #0, CPUs: #1 #2 #3 [ 0.060835] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.237849] node 0 deferred pages initialised in 172ms [ 0.241134] devtmpfs: initialized [ 0.242240] x86/mm: Memory block size: 128MB [ 0.244738] gcov: version magic: 0x41383552 [ 0.246369] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.250109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.253390] pinctrl core: initialized pinctrl subsystem [ 0.255239] [ 0.255778] ************************************************************* [ 0.258018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261016] ** ** [ 0.263015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.265015] ** ** [ 0.267018] ** This means that this kernel is built to expose internal ** [ 0.269022] ** IOMMU data structures, which may compromise security on ** [ 0.272019] ** your system. ** [ 0.274013] ** ** [ 0.276016] ** If you see this message and you are not debugging the ** [ 0.278016] ** kernel, report this immediately to your vendor! ** [ 0.281014] ** ** [ 0.283014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.286020] ************************************************************* [ 0.288679] NET: Registered protocol family 16 [ 0.291498] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.295092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.298091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.302570] cpuidle: using governor menu [ 0.303576] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.306508] PCI: Using configuration type 1 for base access [ 0.308161] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.320073] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.322059] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.327051] cryptd: max_cpu_qlen set to 1000 [ 0.329303] ACPI: Added _OSI(Module Device) [ 0.331016] ACPI: Added _OSI(Processor Device) [ 0.333016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.335026] ACPI: Added _OSI(Processor Aggregator Device) [ 0.339146] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.346433] ACPI: Interpreter enabled [ 0.348065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.349012] ACPI: Using IOAPIC for interrupt routing [ 0.351097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.354845] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.365236] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.368044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.371020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.374077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.379498] acpiphp: Slot [2] registered [ 0.382179] acpiphp: Slot [5] registered [ 0.383109] acpiphp: Slot [6] registered [ 0.385241] acpiphp: Slot [7] registered [ 0.387221] acpiphp: Slot [8] registered [ 0.389195] acpiphp: Slot [9] registered [ 0.390168] acpiphp: Slot [10] registered [ 0.392200] acpiphp: Slot [3] registered [ 0.394176] acpiphp: Slot [4] registered [ 0.396163] acpiphp: Slot [11] registered [ 0.398157] acpiphp: Slot [12] registered [ 0.399183] acpiphp: Slot [13] registered [ 0.401131] acpiphp: Slot [14] registered [ 0.402123] acpiphp: Slot [15] registered [ 0.404140] acpiphp: Slot [16] registered [ 0.406161] acpiphp: Slot [17] registered [ 0.407148] acpiphp: Slot [18] registered [ 0.409149] acpiphp: Slot [19] registered [ 0.411201] acpiphp: Slot [20] registered [ 0.413192] acpiphp: Slot [21] registered [ 0.415112] acpiphp: Slot [22] registered [ 0.416114] acpiphp: Slot [23] registered [ 0.418104] acpiphp: Slot [24] registered [ 0.419090] acpiphp: Slot [25] registered [ 0.421092] acpiphp: Slot [26] registered [ 0.422102] acpiphp: Slot [27] registered [ 0.423101] acpiphp: Slot [28] registered [ 0.425105] acpiphp: Slot [29] registered [ 0.426129] acpiphp: Slot [30] registered [ 0.428106] acpiphp: Slot [31] registered [ 0.429078] PCI host bridge to bus 0000:00 [ 0.431020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.433024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.435025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.438054] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.441028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.443024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.445187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.449218] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.452546] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.463929] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.470056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.473032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.475029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.479055] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.484580] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.486799] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.490048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.492833] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.498015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.512878] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.517805] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.523605] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.531024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.539025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.560027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.572497] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.585017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.592018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.607017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.617640] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.624024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.632023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.653028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.663632] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.668014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.674061] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.689017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.700176] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.707027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.715016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.744015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.757957] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.769100] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.780017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.804015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.820820] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.825417] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.828420] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.830391] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.832217] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.836167] iommu: Default domain type: Passthrough [ 0.838500] SCSI subsystem initialized [ 0.840175] ACPI: bus type USB registered [ 0.842152] usbcore: registered new interface driver usbfs [ 0.845101] usbcore: registered new interface driver hub [ 0.847127] usbcore: registered new device driver usb [ 0.850192] pps_core: LinuxPPS API ver. 1 registered [ 0.852014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.855059] PTP clock support registered [ 0.857157] EDAC MC: Ver: 3.0.0 [ 0.859177] PCI: Using ACPI for IRQ routing [ 0.860445] NetLabel: Initializing [ 0.862017] NetLabel: domain hash size = 128 [ 0.863010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.865087] NetLabel: unlabeled traffic allowed by default [ 0.867153] vgaarb: loaded [ 0.869298] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.871015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.879347] clocksource: Switched to clocksource kvm-clock [ 0.988539] VFS: Disk quotas dquot_6.6.0 [ 0.990530] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.993466] *** VALIDATE ramfs *** [ 0.994912] *** VALIDATE hugetlbfs *** [ 0.996896] pnp: PnP ACPI init [ 1.000323] pnp: PnP ACPI: found 6 devices [ 1.018465] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.022187] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.024523] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.027531] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.031065] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.033775] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.037051] NET: Registered protocol family 2 [ 1.039922] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.047600] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.051773] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.057586] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.060822] TCP: Hash tables configured (established 65536 bind 65536) [ 1.064064] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.067551] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.070635] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.073772] NET: Registered protocol family 1 [ 1.076323] RPC: Registered named UNIX socket transport module. [ 1.078977] RPC: Registered udp transport module. [ 1.080482] RPC: Registered tcp transport module. [ 1.082347] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.084571] NET: Registered protocol family 44 [ 1.086460] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.088767] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.091081] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.093667] PCI: CLS 0 bytes, default 64 [ 1.095593] Unpacking initramfs... [ 2.497682] debug: unmapping init [mem 0xffff9793fcc54000-0xffff9793fffbffff] [ 2.502683] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.507459] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.510731] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.005418] Initialise system trusted keyrings [ 3.007130] Key type blacklist registered [ 3.008932] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.018579] zbud: loaded [ 3.021983] *** VALIDATE nfs *** [ 3.023922] *** VALIDATE nfs4 *** [ 3.025814] pstore: using deflate compression [ 3.029040] Platform Keyring initialized [ 3.134555] NET: Registered protocol family 38 [ 3.136284] Key type asymmetric registered [ 3.137618] Asymmetric key parser 'x509' registered [ 3.138852] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.141330] io scheduler mq-deadline registered [ 3.143551] io scheduler kyber registered [ 3.145539] io scheduler bfq registered [ 3.147096] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.149408] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.151529] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.153627] ACPI: Power Button [PWRF] [ 3.159178] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.165646] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.184552] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.194180] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.207749] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.236731] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.266901] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.272242] Non-volatile memory driver v1.3 [ 3.274213] Linux agpgart interface v0.103 [ 3.305208] virtio_blk virtio1: [vda] 136528 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.308061] vda: detected capacity change from 0 to 69902336 [ 3.324493] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.327622] vdb: detected capacity change from 0 to 1073741824 [ 3.343597] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.347382] vdc: detected capacity change from 0 to 2621440000 [ 3.365643] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.370621] vdd: detected capacity change from 0 to 2621440000 [ 3.386573] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.389629] vde: detected capacity change from 0 to 4294967296 [ 3.404879] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.408531] vdf: detected capacity change from 0 to 4294967296 [ 3.417232] libphy: Fixed MDIO Bus: probed [ 3.424646] usbcore: registered new interface driver usbserial_generic [ 3.427300] usbserial: USB Serial support registered for generic [ 3.430037] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.433916] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.435553] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.437980] mousedev: PS/2 mouse device common for all mice [ 3.441360] rtc_cmos 00:05: RTC can wake from S4 [ 3.443439] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.447798] rtc_cmos 00:05: registered as rtc0 [ 3.452672] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.454132] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.455595] intel_pstate: CPU model not supported [ 3.459760] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.463418] hid: raw HID events driver (C) Jiri Kosina [ 3.465408] usbcore: registered new interface driver usbhid [ 3.467442] usbhid: USB HID core driver [ 3.469209] drop_monitor: Initializing network drop monitor service [ 3.471221] Initializing XFRM netlink socket [ 3.472929] NET: Registered protocol family 10 [ 3.476280] Segment Routing with IPv6 [ 3.477795] NET: Registered protocol family 17 [ 3.480189] mpls_gso: MPLS GSO support [ 3.486717] RAS: Correctable Errors collector initialized. [ 3.489673] AVX version of gcm_enc/dec engaged. [ 3.491736] AES CTR mode by8 optimization enabled [ 3.567502] sched_clock: Marking stable (3567439845, 0)->(4477592230, -910152385) [ 3.571555] registered taskstats version 1 [ 3.574051] Loading compiled-in X.509 certificates [ 3.576548] zswap: loaded using pool lzo/zbud [ 3.603067] Key type big_key registered [ 3.614962] Key type encrypted registered [ 3.616839] ima: No TPM chip found, activating TPM-bypass! [ 3.619401] ima: Allocated hash algorithm: sha1 [ 3.621480] ima: No architecture policies found [ 3.623744] evm: Initialising EVM extended attributes: [ 3.625770] evm: security.selinux [ 3.627259] evm: security.ima [ 3.628490] evm: security.capability [ 3.630061] evm: HMAC attrs: 0x1 [ 3.633189] rtc_cmos 00:05: setting system clock to 2026-06-12 01:41:12 UTC (1781228472) [ 3.639412] debug: unmapping init [mem 0xffffffffa3a03000-0xffffffffa3bfffff] [ 3.643433] debug: unmapping init [mem 0xffffffffa2782000-0xffffffffa2a58fff] [ 3.652185] Write protecting the kernel read-only data: 28672k [ 3.654910] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 3.657916] debug: unmapping init [mem 0xffffffffa1714000-0xffffffffa17fffff] [ 3.697102] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.705457] systemd[1]: Detected virtualization kvm. [ 3.707723] systemd[1]: Detected architecture x86-64. [ 3.709927] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.735025] systemd[1]: No hostname configured. [ 3.738087] systemd[1]: Set hostname to . [ 3.740339] random: systemd: uninitialized urandom read (16 bytes read) [ 3.743268] systemd[1]: Initializing machine ID from random generator. [ 3.792069] random: ln: uninitialized urandom read (6 bytes read) [ 3.886752] random: systemd: uninitialized urandom read (16 bytes read) [ 3.889930] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.895513] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.901125] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.516650] device-mapper: uevent: version 1.0.3 [ 4.519899] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.267926] virtio_net virtio0 ens2: renamed from eth0 [ 5.290888] random: fast init done [ 5.291943] scsi host0: ata_piix [ 5.349492] scsi host1: ata_piix [ 5.350949] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.352492] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.983450] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.135608] random: crng init done [ 10.137335] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.486420] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.554911] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.804568] SELinux: Disabled at runtime. [ 11.868185] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.877372] systemd[1]: Detected virtualization kvm. [ 11.879394] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.362774] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.366214] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.372084] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.375118] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.377439] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.384175] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.391498] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 12.555049] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.762871] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.113385] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.138375] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.246194] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.275311] EDAC sbridge: Ver: 1.1.2 [ 14.490555] Key type dns_resolver registered [ 14.796227] NFS: Registering the id_resolver key type [ 14.798116] Key type id_resolver registered [ 14.799760] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg358-server login: [ 41.520800] libcfs: loading out-of-tree module taints kernel. [ 41.544512] Key type ._llcrypt registered [ 41.546377] Key type .llcrypt registered [ 41.602238] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 50.247470] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 50.902733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.912072] alg: No test for adler32 (adler32-zlib) [ 51.987202] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 52.393814] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 54.031213] Key type lgssc registered [ 54.838167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 65.329019] hrtimer: interrupt took 10011902 ns [ 71.942710] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 118.676497] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 133.176224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 133.207691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.722152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 134.814170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 134.977975] Lustre: lustre-MDT0000: new disk, initializing [ 135.132755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 135.146858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 139.675503] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.615431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.735243] Lustre: 6500:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 154.796138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 154.807355] Lustre: Skipped 1 previous similar message [ 154.957727] Lustre: lustre-MDT0001: new disk, initializing [ 155.102656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 155.172037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 155.190399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.186812] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.978457] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 175.184236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 175.628134] Lustre: lustre-OST0000: new disk, initializing [ 175.636229] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 175.656334] Lustre: 8403:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 175.738056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.923450] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.384338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 185.392899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 185.519482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.348343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.591536] Lustre: lustre-OST0001: new disk, initializing [ 197.598320] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 197.609460] Lustre: 9460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.710205] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.907680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 204.935056] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 205.020988] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 205.695534] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.207404] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.424635] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.800983] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing check_logdir /tmp/testlogs/ [ 239.735251] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing yml_node [ 244.428260] Lustre: DEBUG MARKER: Client: 2.17.53.64 [ 247.248303] Lustre: DEBUG MARKER: MDS: 2.17.53.64 [ 250.040677] Lustre: DEBUG MARKER: OSS: 2.17.53.64 [ 252.340543] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 21:45:18 EDT 2026 [ 276.000643] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 277.718620] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 297.442293] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.449840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 297.460853] Lustre: Skipped 1 previous similar message [ 297.484023] Lustre: Skipped 3 previous similar messages [ 300.377873] Lustre: server umount lustre-MDT0000 complete [ 304.050269] LustreError: 6489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228772 with bad export cookie 10661827691950642820 [ 304.052384] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 304.070708] LustreError: 6489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 307.681970] LustreError: 7440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.684789] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.687580] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.720376] LustreError: 7440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 307.738644] Lustre: Skipped 3 previous similar messages [ 309.728548] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.728905] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.752804] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 309.762288] Lustre: Skipped 2 previous similar messages [ 310.417151] Lustre: server umount lustre-MDT0001 complete [ 315.822566] Lustre: server umount lustre-OST0000 complete [ 320.883039] Lustre: server umount lustre-OST0001 complete [ 328.176841] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 339.102607] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 380.996123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 381.453089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 381.492129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 381.639230] Lustre: lustre-MDT0000: new disk, initializing [ 381.756634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 381.789352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 386.391206] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 397.656568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.852509] Lustre: 14977:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 397.912117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 397.916241] Lustre: Skipped 1 previous similar message [ 398.055472] Lustre: lustre-MDT0001: new disk, initializing [ 398.256407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 398.306339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 398.319141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 403.225874] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 411.553175] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 415.919919] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 423.195046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.553748] Lustre: lustre-OST0000: new disk, initializing [ 423.559668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 423.566237] Lustre: 16724:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 423.701213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.553320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 425.579393] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 425.684369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 430.904492] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 438.493598] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 443.088765] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 443.371926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 448.659857] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 448.909534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 451.556938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.561846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.584896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.596701] LustreError: Skipped 1 previous similar message [ 451.615946] Lustre: Skipped 1 previous similar message [ 456.686866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.702808] Lustre: Skipped 1 previous similar message [ 465.375160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 465.683246] Lustre: server umount lustre-OST0000 complete [ 469.992385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 470.000515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.028155] Lustre: Skipped 1 previous similar message [ 470.037274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 470.044930] Lustre: Skipped 2 previous similar messages [ 476.785213] Lustre: server umount lustre-MDT0000 complete [ 477.153638] LustreError: 15842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.174862] LustreError: 15842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 480.704247] LustreError: 14971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228949 with bad export cookie 10661827691950644360 [ 480.705647] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 480.724268] LustreError: 14971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 480.768208] LustreError: 3650:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff97934311fc00 x1867753477784320/t0(0) o41->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 0 dl 1781228954 ref 1 fl Rpc:RESQU/200/0 rc -5/-107 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 481.111926] Lustre: server umount lustre-MDT0001 complete [ 487.491774] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:49:13 (1781228953) [ 507.625703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.072520] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.166962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 512.810631] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 513.510923] LustreError: 19454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.395035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.219990] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 535.137421] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 539.838076] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 542.270519] Lustre: Failing over lustre-MDT0000 [ 542.480771] Lustre: server umount lustre-MDT0000 complete [ 543.201204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 543.214664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 543.214676] Lustre: Skipped 1 previous similar message [ 543.215588] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.215595] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 559.393340] Lustre: 3651:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781229012/real 1781229012] req@ffff979477591a40 x1867753477799552/t0(0) o400->MGC192.168.203.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781229028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 559.424652] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 559.468083] LustreError: 19454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.506867] LustreError: 19454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 563.869069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.106584] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 575.463681] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 575.489874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 575.522570] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 583.998428] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 588.273029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.284910] Lustre: Skipped 1 previous similar message [ 588.293590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.304106] Lustre: Skipped 3 previous similar messages [ 594.643614] Lustre: server umount lustre-MDT0000 complete [ 597.479374] LustreError: 20662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.523090] LustreError: 20662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 598.481799] LustreError: 19433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229067 with bad export cookie 10661827691950645774 [ 598.486678] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.493602] LustreError: 19433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 598.769346] Lustre: server umount lustre-MDT0001 complete [ 607.228946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.776367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.779073] Lustre: Skipped 2 previous similar messages [ 612.398676] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 621.630886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.296177] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 633.412637] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 637.194410] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 639.419897] Lustre: Failing over lustre-MDT0000 [ 639.744916] Lustre: server umount lustre-MDT0000 complete [ 642.528788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.549147] Lustre: Skipped 2 previous similar messages [ 658.853219] Lustre: 3648:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781229111/real 1781229111] req@ffff979342f9a940 x1867753477828992/t0(0) o400->MGC192.168.203.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781229127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 658.907790] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.358589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.032824] LustreError: 22454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.048135] LustreError: 22454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 669.152155] LustreError: 3647:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff979342f9a940 x1867753477832064/t0(0) o250->MGC192.168.203.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 669.680311] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 674.337816] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 674.792133] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 674.817261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 674.834939] Lustre: Skipped 1 previous similar message [ 674.855194] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 682.991422] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 690.147763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.148640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 690.162736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 690.184811] Lustre: Skipped 3 previous similar messages [ 693.528209] Lustre: server umount lustre-MDT0000 complete [ 697.270098] LustreError: 22433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229166 with bad export cookie 10661827691950646712 [ 697.271802] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 697.281253] LustreError: 22433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 697.745718] Lustre: server umount lustre-MDT0001 complete [ 707.312523] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:52:53 (1781229173) [ 725.282942] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 734.968782] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 781.336154] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 794.137854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.532995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 794.613730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 794.763804] Lustre: lustre-MDT0000: new disk, initializing [ 794.893745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.901721] Lustre: Skipped 2 previous similar messages [ 794.924949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 800.765407] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 813.396644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 813.552186] Lustre: 28439:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 814.101227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 819.945982] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 828.973137] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 833.109771] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 840.434932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 840.784244] Lustre: 30180:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 842.694213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 842.747484] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 842.822445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 842.842926] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 847.306226] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 856.169933] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 865.367385] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 888.806167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.808512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 888.809538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 888.809545] Lustre: Skipped 2 previous similar messages [ 888.823570] Lustre: Skipped 2 previous similar messages [ 888.873866] LustreError: Skipped 1 previous similar message [ 890.368685] Lustre: server umount lustre-OST0000 complete [ 902.802695] Lustre: server umount lustre-MDT0000 complete [ 904.164294] LustreError: 28451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.182872] LustreError: 28451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 906.395576] LustreError: 30179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229375 with bad export cookie 10661827691950647118 [ 906.406786] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 906.414688] LustreError: 30179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 907.137897] Lustre: server umount lustre-MDT0001 complete [ 918.962896] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 922.976025] Key type lgssc unregistered [ 923.410575] LNet: 32031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.422420] LNetError: 32031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.447309] LNet: Removed LNI 192.168.203.158@tcp [ 924.676498] Key type .llcrypt unregistered [ 924.681541] Key type ._llcrypt unregistered [ 951.871885] Key type ._llcrypt registered [ 951.877219] Key type .llcrypt registered [ 951.994984] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 953.625668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 953.635477] alg: No test for adler32 (adler32-zlib) [ 954.862153] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 955.134899] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 956.895202] Key type lgssc registered [ 958.227715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 966.526815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 966.542416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.053341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 973.218764] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 977.382947] LustreError: 33428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 977.398030] LustreError: 33428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 982.373803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.500251] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 982.685203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 986.655280] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 992.690822] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 998.233824] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1006.589391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1007.266702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1008.338393] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1014.450677] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1020.306713] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1036.300764] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:58:22 (1781229502) [ 1038.298764] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1040.258394] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1047.136907] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:58:33 (1781229513) [ 1055.841651] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:58:42 (1781229522) [ 1056.788548] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1065.937180] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1065.949696] Lustre: Skipped 2 previous similar messages [ 1067.829745] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:58:54 (1781229534) [ 1071.312431] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1071.317256] Lustre: Skipped 3 previous similar messages [ 1091.317831] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1091.328629] Lustre: Skipped 6 previous similar messages [ 1094.372966] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:59:20 (1781229560) [ 1097.111641] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1100.396857] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1109.105712] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:59:35 (1781229575) [ 1110.863840] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1110.869718] Lustre: Skipped 3 previous similar messages [ 1122.997886] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:59:49 (1781229589) [ 1140.708871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1140.709856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.710721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1140.759093] Lustre: Skipped 1 previous similar message [ 1145.848567] Lustre: server umount lustre-OST0000 complete [ 1150.957923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.965787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1150.978246] Lustre: Skipped 1 previous similar message [ 1151.005524] Lustre: Skipped 2 previous similar messages [ 1155.546408] Lustre: server umount lustre-MDT0000 complete [ 1156.067908] LustreError: 35108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.091229] LustreError: 35108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1159.775114] LustreError: 37621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229628 with bad export cookie 14491983652644616800 [ 1159.781429] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.794430] LustreError: 37621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1160.262955] Lustre: server umount lustre-MDT0001 complete [ 1171.478878] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1174.794220] Key type lgssc unregistered [ 1175.077730] LNet: 40936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.092272] LNetError: 40936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.117134] LNet: Removed LNI 192.168.203.158@tcp [ 1175.904935] Key type .llcrypt unregistered [ 1175.907316] Key type ._llcrypt unregistered [ 1191.881296] Key type ._llcrypt registered [ 1191.884846] Key type .llcrypt registered [ 1192.001216] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 1209.155986] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1210.117576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1210.151049] alg: No test for adler32 (adler32-zlib) [ 1211.197487] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 1211.489590] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1213.200977] Key type lgssc registered [ 1214.409660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1253.382468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1253.417908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1254.717957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1254.746805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1254.834532] Lustre: lustre-MDT0000: new disk, initializing [ 1254.908882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1254.935061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1259.517891] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1269.576122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.658260] Lustre: 44065:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1269.701723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1269.705898] Lustre: Skipped 1 previous similar message [ 1269.804187] Lustre: lustre-MDT0001: new disk, initializing [ 1269.911804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1269.951228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1269.967701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1274.601233] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1282.288534] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1286.223370] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1292.622328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1292.886942] Lustre: lustre-OST0000: new disk, initializing [ 1292.891511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1292.901401] Lustre: 45809:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1292.989205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1299.935450] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1302.073908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1302.081284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1302.215106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1307.339456] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1312.293215] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1312.493264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1316.574581] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1316.782086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1322.982568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.986860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1323.024678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1323.035214] Lustre: Skipped 1 previous similar message [ 1324.796655] Lustre: server umount lustre-OST0000 complete [ 1331.684064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1331.698975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.713576] Lustre: Skipped 1 previous similar message [ 1331.723697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1333.219637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1333.223411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.717413] Lustre: server umount lustre-MDT0000 complete [ 1339.097329] LustreError: 45808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229807 with bad export cookie 1751774550863407912 [ 1339.108468] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1339.126903] LustreError: 45808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1339.521535] Lustre: server umount lustre-MDT0001 complete [ 1351.913495] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:03:38 (1781229818) [ 1359.022811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.377464] LustreError: 48122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.404661] LustreError: 48122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1359.501431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.875748] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1364.963410] LustreError: 48123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1370.082968] LustreError: 48122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.948454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.418269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1379.410503] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1385.744397] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1389.656690] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1397.326258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1397.585702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.449033] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1412.051544] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1417.341263] LustreError: 50310:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1418.392864] Lustre: Modifying parameter general.timeout=1 in log params [ 1420.004603] Lustre: Modifying parameter general.timeout=3 in log params [ 1420.014532] Lustre: Skipped 1 previous similar message [ 1422.503548] Lustre: Modifying parameter general.timeout=6 in log params [ 1422.507964] Lustre: Skipped 2 previous similar messages [ 1427.336658] Lustre: Modifying parameter general.timeout=12 in log params [ 1427.341782] Lustre: Skipped 5 previous similar messages [ 1436.001566] Lustre: Modifying parameter general.timeout=23 in log params [ 1436.005791] Lustre: Skipped 10 previous similar messages [ 1452.421335] Lustre: Modifying parameter general.timeout=41 in log params [ 1452.433639] Lustre: Skipped 17 previous similar messages [ 1485.398848] Lustre: Modifying parameter general.timeout=75 in log params [ 1485.402915] Lustre: Skipped 33 previous similar messages [ 1547.924333] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:06:53 (1781230013) [ 1550.057303] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1550.061638] Lustre: Skipped 56 previous similar messages [ 1551.052473] Lustre: 57253:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1553.182836] Lustre: 57352:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1555.354945] Lustre: 57450:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1561.144179] Lustre: 57764:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1561.158585] Lustre: 57764:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1586.207779] Lustre: 59121:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1586.242048] LustreError: 59121:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1590.487663] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:07:36 (1781230056) [ 1604.064884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1604.075344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.094913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1604.103260] Lustre: Skipped 1 previous similar message [ 1609.514631] Lustre: server umount lustre-MDT0000 complete [ 1612.770465] LustreError: 48122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1612.786277] LustreError: 48122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1613.192531] LustreError: 52838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230082 with bad export cookie 1751774550863408794 [ 1613.193569] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1613.205843] LustreError: 52838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1613.566297] Lustre: server umount lustre-MDT0001 complete [ 1627.981951] Lustre: server umount lustre-OST0000 complete [ 1655.741474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1656.035253] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1656.312804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.298265] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1669.112163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.294603] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1669.357883] Lustre: 61654:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1674.235522] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1680.029709] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1684.388303] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1690.911300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1691.054561] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1691.300435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.304501] Lustre: Skipped 1 previous similar message [ 1698.135200] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1703.265591] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1705.034958] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1705.041736] Lustre: Skipped 21 previous similar messages [ 1705.091112] Lustre: 63880:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1715.168512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1715.184630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.215385] Lustre: Skipped 2 previous similar messages [ 1715.229603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1715.238418] Lustre: Skipped 3 previous similar messages [ 1719.317752] Lustre: server umount lustre-OST0000 complete [ 1725.922542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.949426] Lustre: Skipped 1 previous similar message [ 1725.978546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1725.981578] Lustre: Skipped 1 previous similar message [ 1737.183941] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1737.408483] Lustre: server umount lustre-MDT0000 complete [ 1740.264224] LustreError: 62569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1740.286707] LustreError: 62569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1741.195263] LustreError: 63288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230210 with bad export cookie 1751774550863411797 [ 1741.205812] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1741.213191] LustreError: 63288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1741.606860] Lustre: server umount lustre-MDT0001 complete [ 1752.617512] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1754.961619] Key type lgssc unregistered [ 1755.225346] LNet: 65076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.242411] LNetError: 65076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.259112] LNet: Removed LNI 192.168.203.158@tcp [ 1756.136516] Key type .llcrypt unregistered [ 1756.141472] Key type ._llcrypt unregistered [ 1783.361576] Key type ._llcrypt registered [ 1783.363611] Key type .llcrypt registered [ 1783.445891] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1784.139937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1784.237739] alg: No test for adler32 (adler32-zlib) [ 1785.482917] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 1785.832207] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1787.495361] Key type lgssc registered [ 1788.822722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1796.347762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1796.371594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.850979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.335401] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1807.331668] LustreError: 66479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.352065] LustreError: 66479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1811.000619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.593786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1816.183429] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1822.091560] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1825.775133] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1832.549381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1833.240683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1839.445855] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1845.165290] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1853.436856] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1862.624521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1862.631122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.641939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1863.141671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.147420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1863.164452] Lustre: Skipped 1 previous similar message [ 1863.175193] Lustre: Skipped 2 previous similar messages [ 1867.262306] Lustre: server umount lustre-MDT0000 complete [ 1868.267157] LustreError: 66475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1868.312961] LustreError: 66475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1871.386931] LustreError: 66459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230340 with bad export cookie 7869400621554056186 [ 1871.387995] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.400396] LustreError: 66459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1871.878090] Lustre: server umount lustre-MDT0001 complete [ 1886.957789] Lustre: server umount lustre-OST0000 complete [ 1915.780720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.147938] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1916.208592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1916.559234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.813831] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1930.124842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.239185] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1930.300973] Lustre: 71012:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1931.015088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.003721] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1941.116428] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1946.387272] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1953.765697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1954.040843] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1954.636656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.802746] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1968.433572] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1970.421418] Lustre: 73248:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1972.768352] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1979.363964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.366515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1979.386602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1984.499050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1984.510770] Lustre: Skipped 1 previous similar message [ 1985.091864] Lustre: server umount lustre-OST0000 complete [ 1989.606704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.636721] Lustre: Skipped 1 previous similar message [ 1989.647210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1989.659075] Lustre: Skipped 2 previous similar messages [ 1995.754198] Lustre: server umount lustre-MDT0000 complete [ 1999.841151] LustreError: 71018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.883717] LustreError: 71018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2000.464300] LustreError: 71003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230469 with bad export cookie 7869400621554057026 [ 2000.484680] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2000.924568] Lustre: server umount lustre-MDT0001 complete [ 2013.553842] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2016.418292] Key type lgssc unregistered [ 2016.816074] LNet: 74443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.826909] LNetError: 74443:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.838731] LNet: Removed LNI 192.168.203.158@tcp [ 2017.699625] Key type .llcrypt unregistered [ 2017.702267] Key type ._llcrypt unregistered [ 2045.936731] Key type ._llcrypt registered [ 2045.939701] Key type .llcrypt registered [ 2046.041100] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2047.033817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2047.088838] alg: No test for adler32 (adler32-zlib) [ 2048.088728] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 2048.265916] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2049.911220] Key type lgssc registered [ 2051.187884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2058.815088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2058.838915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.261776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.477085] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2070.502498] LustreError: 75844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2070.524765] LustreError: 75844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2072.223365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.826830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2077.436113] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2083.179029] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2087.449317] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2094.638896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2095.325461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.588236] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2108.144903] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2114.365995] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2115.283613] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2117.143370] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2117.147257] Lustre: Skipped 1 previous similar message [ 2119.919945] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2119.922870] Lustre: Skipped 2 previous similar messages [ 2124.712480] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2124.723713] Lustre: Skipped 3 previous similar messages [ 2133.255852] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2133.266056] Lustre: Skipped 8 previous similar messages [ 2149.455290] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2149.463276] Lustre: Skipped 15 previous similar messages [ 2182.015899] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2182.021770] Lustre: Skipped 34 previous similar messages [ 2216.730829] Lustre: 83604:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2219.705068] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:18:06 (1781230686) [ 2221.494574] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2223.351886] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:18:09 (1781230689) [ 2228.711164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2228.724808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.729439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2228.761402] Lustre: Skipped 1 previous similar message [ 2228.778747] Lustre: Skipped 1 previous similar message [ 2233.870927] LustreError: 77446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.897619] LustreError: 77446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2233.975479] Lustre: server umount lustre-OST0000 complete [ 2238.947569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.975745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2238.982231] Lustre: Skipped 1 previous similar message [ 2244.027398] Lustre: server umount lustre-MDT0000 complete [ 2244.067612] LustreError: 75844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.557147] LustreError: 75824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230717 with bad export cookie 2010796440162504099 [ 2248.570793] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2248.573031] LustreError: 75824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2249.068052] Lustre: server umount lustre-MDT0001 complete [ 2260.374489] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2263.308302] Key type lgssc unregistered [ 2263.587463] LNet: 84838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.592473] LNetError: 84838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.616189] LNet: Removed LNI 192.168.203.158@tcp [ 2264.390396] Key type .llcrypt unregistered [ 2264.393874] Key type ._llcrypt unregistered [ 2268.314293] Key type ._llcrypt registered [ 2268.318400] Key type .llcrypt registered [ 2268.422229] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2272.305375] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules [ 2273.324601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2273.381266] alg: No test for adler32 (adler32-zlib) [ 2274.470344] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 2274.751607] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2276.447188] Key type lgssc registered [ 2277.706646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2279.271827] LustreError: 86023:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2299.894507] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2301.932489] LustreError: 86023:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2309.156834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2309.180578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.786918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2314.631335] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2320.371897] LustreError: 87032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.387801] LustreError: 87032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2322.129545] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:19:48 (1781230788) [ 2325.479248] LustreError: 87031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.261084] LustreError: 87063:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2326.277667] LustreError: 87063:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2326.287408] LustreError: 87063:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2326.710443] Lustre: server umount lustre-MDT0000 complete [ 2339.350429] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2342.182899] Key type lgssc unregistered [ 2342.488851] LNet: 88521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2342.503846] LNetError: 88521:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2342.518870] LNet: Removed LNI 192.168.203.158@tcp [ 2343.288545] Key type .llcrypt unregistered [ 2343.297368] Key type ._llcrypt unregistered [ 2361.798130] Key type ._llcrypt registered [ 2361.802630] Key type .llcrypt registered [ 2361.934846] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2363.203612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2363.233878] alg: No test for adler32 (adler32-zlib) [ 2364.383283] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 2364.674841] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2366.343384] Key type lgssc registered [ 2367.537845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2375.948800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2375.974677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.694375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.473977] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2387.942813] LustreError: 89727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.965254] LustreError: 89727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2392.305085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2393.353420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2398.081763] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2403.949204] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2407.556505] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2414.832242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2415.579874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2421.585748] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2427.622425] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2478.844266] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:22:25 (1781230945) [ 2480.704694] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2482.688890] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:22:29 (1781230949) [ 2489.314800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.316996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2489.334649] Lustre: Skipped 2 previous similar messages [ 2489.373501] Lustre: Skipped 2 previous similar messages [ 2493.798777] Lustre: server umount lustre-MDT0000 complete [ 2494.435608] LustreError: 89728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.460327] LustreError: 89728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2498.043830] LustreError: 92651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230966 with bad export cookie 12562002834238248264 [ 2498.064787] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2498.581738] Lustre: server umount lustre-MDT0001 complete [ 2513.794120] Lustre: server umount lustre-OST0000 complete [ 2526.803116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.436228] LustreError: 93681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.645689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.496829] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2532.836248] LustreError: 93682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.956183] LustreError: 93681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2541.789373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.756647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2547.604641] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2553.956358] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2557.443707] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2571.410078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2571.709759] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2571.720902] LustreError: 93671:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2571.760198] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2571.773825] LustreError: 95540:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2571.788870] LustreError: 95540:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2571.801694] LustreError: 95540:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2571.930242] Lustre: server umount lustre-OST0000 complete [ 2571.939329] LustreError: 95540:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2582.290949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2582.559059] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2582.569901] LustreError: 93668:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2582.593046] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2582.600100] LustreError: 96004:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2582.607840] LustreError: 96004:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2582.613587] LustreError: 96004:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2582.681557] Lustre: server umount lustre-OST0000 complete [ 2582.687677] LustreError: 96004:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2592.958416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2593.187894] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2593.198861] Lustre: Found index 0 for lustre-OST0000, updating log [ 2593.209840] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2593.327608] Lustre: lustre-OST0000: new disk, initializing [ 2593.336145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2593.341673] Lustre: 96539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2593.675029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.307347] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2606.249495] Lustre: Failing over lustre-OST0000 [ 2606.635701] Lustre: server umount lustre-OST0000 complete [ 2613.219562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2613.253465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.266124] Lustre: Skipped 1 previous similar message [ 2614.313059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2614.330942] Lustre: Skipped 1 previous similar message [ 2619.360512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2619.383693] Lustre: Skipped 1 previous similar message [ 2620.533766] Lustre: server umount lustre-MDT0000 complete [ 2624.487876] LustreError: 93678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.522498] LustreError: 93678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2624.811847] LustreError: 93661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231093 with bad export cookie 12562002834238251008 [ 2624.823713] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2624.832800] LustreError: 93661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2625.314034] Lustre: server umount lustre-MDT0001 complete [ 2635.555644] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:25:01 (1781231101) [ 2656.976589] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2670.889915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.780152] LustreError: 98857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.983154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.281659] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2687.418626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.110838] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2696.377987] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2705.080799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2705.751617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2705.755937] Lustre: Skipped 1 previous similar message [ 2705.770893] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2712.976203] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2712.987552] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2713.036212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2713.311471] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2717.136592] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 2717.168127] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2717.173809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2717.212130] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2717.226132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2723.324872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.437579] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2723.559801] Lustre: lustre-OST0001: new disk, initializing [ 2723.563446] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2723.568869] Lustre: 101128:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2731.188558] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2732.647396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2732.657242] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2732.762428] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2739.966365] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2750.975874] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2758.629984] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.651490] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2758.678429] Lustre: Skipped 2 previous similar messages [ 2758.702488] Lustre: Skipped 2 previous similar messages [ 2763.207531] Lustre: server umount lustre-MDT0001 complete [ 2763.747638] LustreError: 99571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.773944] LustreError: 99571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2775.247493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.359263] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2775.367527] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2775.438758] Lustre: 98845:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2776.413433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2776.430569] Lustre: Skipped 1 previous similar message [ 2776.653143] LustreError: 102631:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2776.687610] LustreError: 102631:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2776.704693] LustreError: 98849:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2776.829553] LustreError: 102646:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.158@tcp: cfg command failed: rc = -17 [ 2776.838074] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2776.838074] [ 2781.408707] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2781.673036] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2793.114272] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:27:39 (1781231259) [ 2812.129648] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2829.896932] Lustre: 104690:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2856.625759] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2862.348605] Lustre: 105825:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2960.351963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2960.361373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.378883] Lustre: Skipped 1 previous similar message [ 2960.386167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.397781] Lustre: Skipped 1 previous similar message [ 2963.903687] Lustre: server umount lustre-MDT0000 complete [ 2965.991662] LustreError: 105604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.016745] LustreError: 105604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 2967.728295] LustreError: 101127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231436 with bad export cookie 12562002834238252009 [ 2967.731824] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2967.738063] LustreError: 101127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2968.044108] Lustre: server umount lustre-MDT0001 complete [ 2981.465637] Lustre: server umount lustre-OST0000 complete [ 2996.799163] Lustre: server umount lustre-OST0001 complete [ 3003.787434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3017.167739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3026.483746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3037.151429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3059.364869] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3070.420652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.435841] Lustre: lustre-MDT0000: reset Object Index mappings [ 3071.036961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.615662] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3084.577938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.594944] Lustre: lustre-MDT0001: reset Object Index mappings [ 3090.029890] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3093.247644] Lustre: 110881:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3099.939043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3107.814979] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3109.859793] LustreError: 111235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.873605] LustreError: 111235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3109.881230] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3116.080207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3116.366443] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.178046] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3122.185687] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3123.590883] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3134.465784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3139.851449] Lustre: 112723:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3197.192370] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:34:23 (1781231663) [ 3223.525816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3223.542478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.561446] Lustre: Skipped 3 previous similar messages [ 3223.567550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3223.576927] Lustre: Skipped 3 previous similar messages [ 3227.556186] Lustre: server umount lustre-MDT0000 complete [ 3231.294917] LustreError: 109747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231700 with bad export cookie 12562002834238299098 [ 3231.300192] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3231.308885] LustreError: 109747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3234.785460] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.815412] Lustre: Skipped 3 previous similar messages [ 3238.128487] Lustre: server umount lustre-MDT0001 complete [ 3248.905822] Lustre: server umount lustre-OST0000 complete [ 3267.042948] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3267.352439] Lustre: server umount lustre-OST0001 complete [ 3274.593129] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 3285.793376] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3325.134150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.428096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3325.474199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3325.564392] Lustre: lustre-MDT0000: new disk, initializing [ 3325.669920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3330.514698] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3338.555868] Lustre: server umount lustre-MDT0000 complete [ 3348.157872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.430324] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3348.458535] Lustre: 116969:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3348.484385] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3348.730412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.739616] Lustre: Skipped 4 previous similar messages [ 3352.951923] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3357.735639] Lustre: server umount lustre-MDT0000 complete [ 3365.885468] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:37:12 (1781231832) [ 3367.529644] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3369.402509] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:37:15 (1781231835) [ 3385.932958] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3397.068233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3402.342191] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3410.642855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.776461] Lustre: 118989:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3410.817689] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3410.821583] Lustre: Skipped 1 previous similar message [ 3410.898579] Lustre: lustre-MDT0001: new disk, initializing [ 3411.016920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3411.032700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3415.138543] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3420.166683] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3428.109207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3428.359990] Lustre: lustre-OST0000: new disk, initializing [ 3428.362610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3428.367492] Lustre: 120531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3429.585184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3429.590791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3429.726921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3434.619875] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3445.358980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3445.617274] Lustre: lustre-OST0001: new disk, initializing [ 3445.621724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3445.631940] Lustre: 121387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3447.251048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3447.262427] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3447.367211] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3452.040638] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3462.529868] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3466.239170] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3878.772315] Lustre: 127786:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3878.784323] Lustre: 127786:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3878.790434] Lustre: 127786:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3878.798779] Lustre: 127786:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3878.803277] Lustre: 127786:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3878.809264] Lustre: 127786:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4207.064374] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:51:12 (1781232672) [ 4224.996562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4225.000312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.004846] Lustre: Skipped 1 previous similar message [ 4225.007254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4225.009482] Lustre: Skipped 8 previous similar messages [ 4228.832453] Lustre: server umount lustre-MDT0000 complete [ 4230.628965] LustreError: 122216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.647599] LustreError: 122216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 4232.429787] LustreError: 127808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232701 with bad export cookie 12562002834238368587 [ 4232.437290] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4232.454303] LustreError: 127808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4235.752799] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.756107] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4235.782803] Lustre: Skipped 4 previous similar messages [ 4235.787151] Lustre: Skipped 4 previous similar messages [ 4239.094656] Lustre: server umount lustre-MDT0001 complete [ 4258.273084] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4258.385771] Lustre: server umount lustre-OST0000 complete [ 4268.704501] Lustre: server umount lustre-OST0001 complete [ 4276.114566] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 4286.599113] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 4323.551267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.845167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4323.878701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4323.961724] Lustre: lustre-MDT0000: new disk, initializing [ 4324.054245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4324.061378] Lustre: Skipped 4 previous similar messages [ 4324.081907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4329.200447] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4341.189163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.302244] Lustre: 130811:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4341.367635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4341.375355] Lustre: Skipped 1 previous similar message [ 4341.500435] Lustre: lustre-MDT0001: new disk, initializing [ 4341.703151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4341.718528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4346.371896] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4354.313103] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4358.502719] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4365.347973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4365.650687] Lustre: lustre-OST0000: new disk, initializing [ 4365.654267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4365.694822] Lustre: 132553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4367.146990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4367.161402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4367.271768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4372.044312] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4379.958660] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4390.568432] Lustre: lustre-MDD0000: changelog on [ 4393.294496] Lustre: lustre-MDD0001: changelog on [ 6512.170687] Lustre: lustre-MDD0001: changelog off [ 6514.262701] Lustre: 130820:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6515.211885] Lustre: lustre-MDD0000: changelog off [ 6522.856328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6522.857888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6522.860224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6522.860236] Lustre: Skipped 3 previous similar messages [ 6522.875435] LustreError: Skipped 1 previous similar message [ 6522.888798] Lustre: Skipped 1 previous similar message [ 6527.991567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6528.001249] Lustre: Skipped 2 previous similar messages [ 6533.094276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6533.099128] Lustre: Skipped 1 previous similar message [ 6535.135336] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6535.459235] Lustre: server umount lustre-OST0000 complete [ 6543.329289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.349737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6543.354725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6543.386747] Lustre: Skipped 1 previous similar message [ 6553.571279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6553.574203] Lustre: Skipped 3 previous similar messages [ 6555.103139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6555.525786] Lustre: server umount lustre-MDT0000 complete [ 6558.699102] LustreError: 130824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6558.718538] LustreError: 130824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6559.995771] LustreError: 130805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235028 with bad export cookie 12562002834239193614 [ 6560.001152] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6560.018455] LustreError: 130805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6560.427383] Lustre: server umount lustre-MDT0001 complete [ 6572.062440] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 6575.592490] Key type lgssc unregistered [ 6575.881926] LNet: 141595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6575.894613] LNetError: 141595:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6575.915676] LNet: Removed LNI 192.168.203.158@tcp [ 6577.036146] Key type .llcrypt unregistered [ 6577.041159] Key type ._llcrypt unregistered [ 6596.951727] Key type ._llcrypt registered [ 6596.953552] Key type .llcrypt registered [ 6597.081681] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 6613.207784] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6614.233132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6614.300793] alg: No test for adler32 (adler32-zlib) [ 6615.438351] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 6615.697848] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 6617.423223] Key type lgssc registered [ 6618.679282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6655.955338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6655.984208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6657.285426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6657.347711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6657.469408] Lustre: lustre-MDT0000: new disk, initializing [ 6657.565634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6657.597675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6661.926967] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6673.523273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.813047] Lustre: 144784:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6673.892071] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6673.896961] Lustre: Skipped 1 previous similar message [ 6674.003151] Lustre: lustre-MDT0001: new disk, initializing [ 6674.124975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6674.167623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6674.180808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6678.812859] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6686.161288] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6690.163215] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6696.346733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6696.612278] Lustre: lustre-OST0000: new disk, initializing [ 6696.617947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6696.625354] Lustre: 146525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6696.728498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6702.415138] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6706.200040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6706.208674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6706.318075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6709.116400] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6715.497289] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_module obdecho/obdecho [ 6717.159901] LustreError: 147396:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6717.168033] LustreError: 147396:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6718.703512] LustreError: 147492:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6730.726837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6730.732857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.743897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6732.258436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6732.264291] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.267090] Lustre: Skipped 1 previous similar message [ 6732.283210] Lustre: Skipped 1 previous similar message [ 6734.999978] Lustre: server umount lustre-MDT0000 complete [ 6738.504390] LustreError: 144775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235207 with bad export cookie 8693472044171693316 [ 6738.506421] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6738.516351] LustreError: 144775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6739.072919] Lustre: server umount lustre-MDT0001 complete [ 6753.216691] Lustre: server umount lustre-OST0000 complete [ 6772.842457] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 6782.574688] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6816.359433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.614533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6816.642449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6816.710423] Lustre: lustre-MDT0000: new disk, initializing [ 6816.785074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6816.811831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6820.162238] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6830.459626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6830.573738] Lustre: 151176:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6830.617199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6830.620386] Lustre: Skipped 1 previous similar message [ 6830.678830] Lustre: lustre-MDT0001: new disk, initializing [ 6830.781116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6830.828966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6830.840847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6835.833327] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6845.639291] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6850.387642] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6858.963853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6859.276310] Lustre: lustre-OST0000: new disk, initializing [ 6859.283932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6859.289903] Lustre: 152918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6859.377259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6864.924162] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6867.553917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6867.571277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6867.696496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6872.993399] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6876.870143] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6877.074339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6881.529839] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6881.752354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6888.420172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6888.435574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6888.436887] Lustre: Skipped 1 previous similar message [ 6888.439868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6888.461202] Lustre: Skipped 2 previous similar messages [ 6889.817743] Lustre: server umount lustre-OST0000 complete [ 6893.538982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.542527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6893.553056] Lustre: Skipped 1 previous similar message [ 6893.565295] Lustre: Skipped 1 previous similar message [ 6898.656040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6898.674032] Lustre: Skipped 1 previous similar message [ 6899.633320] Lustre: server umount lustre-MDT0000 complete [ 6903.511106] LustreError: 151169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235372 with bad export cookie 8693472044171694275 [ 6903.515087] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6903.526685] LustreError: 151169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6903.961405] Lustre: server umount lustre-MDT0001 complete [ 6907.376483] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:36:13 (1781235373) [ 6908.946921] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6910.688550] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:36:17 (1781235377) [ 6918.433333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.902623] LustreError: 155182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6918.937730] LustreError: 155182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6919.012930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6923.263314] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6924.264318] LustreError: 155183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6928.355872] LustreError: 155182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.290518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6937.253718] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6942.862966] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6946.393652] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6953.146964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6953.501823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6953.509346] Lustre: Skipped 1 previous similar message [ 6959.413711] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6964.740403] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6973.923385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6973.932804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6973.948207] Lustre: Skipped 1 previous similar message [ 6973.966485] Lustre: Skipped 2 previous similar messages [ 6977.465711] Lustre: server umount lustre-MDT0000 complete [ 6979.050733] LustreError: 155182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6979.072768] LustreError: 155182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6981.109114] LustreError: 155164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235449 with bad export cookie 8693472044171695157 [ 6981.118749] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6981.122328] LustreError: 155164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6984.160930] LustreError: 155177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.164891] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.186774] LustreError: 155177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6984.202651] Lustre: Skipped 1 previous similar message [ 6987.661335] Lustre: server umount lustre-MDT0001 complete [ 6994.031250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7007.004719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7007.190106] LustreError: 158688:0:(llog.c:1646:llog_backup()) MGC192.168.203.158@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7007.197411] Lustre: 158688:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.158@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7011.809212] LustreError: 158693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7011.842399] LustreError: 158693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7031.264495] LustreError: 158695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7031.282905] LustreError: 158695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 7067.257734] LustreError: 158693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.270531] LustreError: 158693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 7068.127384] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7068.132736] Lustre: 158688:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.158@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7109.113195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7113.090430] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7114.248749] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7121.226191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7126.563034] LustreError: 143003:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cec184a3 x1867760359563136/t0(0) o250->MGC192.168.203.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7131.213374] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7132.145259] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7136.447580] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7138.118837] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7142.038492] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7144.446824] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7172.557808] Lustre: server umount lustre-MDT0000 complete [ 7173.087969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7173.101978] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7173.109095] LustreError: 156794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7173.122755] Lustre: Skipped 2 previous similar messages [ 7173.155848] LustreError: 156794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 46 previous similar messages [ 7177.053917] LustreError: 156805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235645 with bad export cookie 8693472044171701611 [ 7177.057372] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7177.068603] LustreError: 156805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7177.472118] Lustre: server umount lustre-MDT0001 complete [ 7192.007440] Lustre: server umount lustre-OST0000 complete [ 7214.290419] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 7224.816471] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7262.079877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.324991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7262.348490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7262.436443] Lustre: lustre-MDT0000: new disk, initializing [ 7262.539577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7262.545788] Lustre: Skipped 1 previous similar message [ 7262.559232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7267.535795] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7277.303096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7277.393140] Lustre: 163535:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7277.645652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7282.283193] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7290.096733] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7294.110264] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7300.513987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7300.809079] Lustre: lustre-OST0000: new disk, initializing [ 7300.813943] Lustre: Skipped 1 previous similar message [ 7300.822560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7300.829688] Lustre: Skipped 2 previous similar messages [ 7300.834700] Lustre: 165273:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7306.363250] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7309.410408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7309.420105] Lustre: Skipped 1 previous similar message [ 7309.429237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7309.584326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7313.693967] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7317.722128] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7318.014615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7322.873267] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7323.116225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7330.272833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7330.304042] Lustre: Skipped 1 previous similar message [ 7330.317476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7330.322891] Lustre: Skipped 4 previous similar messages [ 7330.329148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7331.670348] Lustre: server umount lustre-OST0000 complete [ 7338.979114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7342.910959] Lustre: server umount lustre-MDT0000 complete [ 7345.632729] LustreError: 163545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7345.658728] LustreError: 163545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7347.139313] LustreError: 164311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235816 with bad export cookie 8693472044171718439 [ 7347.151572] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7347.154799] LustreError: 164311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7347.536553] Lustre: server umount lustre-MDT0001 complete [ 7358.347208] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:43:44 (1781235824) [ 7366.333229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.524270] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7380.288091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7384.843312] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7391.274687] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7395.523650] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7402.970910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7409.978944] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7415.338815] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7426.231407] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:44:52 (1781235892) [ 7432.163058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7432.175770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.180038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7432.192107] Lustre: Skipped 3 previous similar messages [ 7432.215225] Lustre: Skipped 4 previous similar messages [ 7437.699076] Lustre: server umount lustre-OST0000 complete [ 7447.981717] Lustre: server umount lustre-MDT0000 complete [ 7451.567181] LustreError: 167520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235920 with bad export cookie 8693472044171719321 [ 7451.573834] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7451.577982] LustreError: 167520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7451.912624] Lustre: server umount lustre-MDT0001 complete [ 7460.310307] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7462.384493] Key type lgssc unregistered [ 7462.661093] LNet: 170906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7462.665949] LNetError: 170906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7462.682890] LNet: Removed LNI 192.168.203.158@tcp [ 7463.256388] Key type .llcrypt unregistered [ 7463.258167] Key type ._llcrypt unregistered [ 7478.156595] Key type ._llcrypt registered [ 7478.159534] Key type .llcrypt registered [ 7478.257636] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7479.187842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7479.248548] alg: No test for adler32 (adler32-zlib) [ 7480.237268] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 7480.462777] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 7482.128404] Key type lgssc registered [ 7483.072845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7490.296551] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7490.308287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7505.889196] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7511.007240] LustreError: 172131:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.158@tcp: failed processing log, type 4: rc = -110 [ 7516.193618] LustreError: 172131:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7516.221548] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7516.231101] LustreError: MGC192.168.203.158@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7516.269715] LustreError: 172131:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7516.292055] LustreError: 172131:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7516.323209] LustreError: 172131:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7516.410739] Lustre: server umount lustre-OST0000 complete [ 7516.416723] LustreError: 172131:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7525.165109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7526.609341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7530.533257] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7537.121134] LustreError: 172630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7537.157158] LustreError: 172630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7539.944331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7540.288861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7544.356698] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7550.084443] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7554.434719] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7561.783572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7561.977583] LustreError: 174222:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7561.992633] Lustre: 174222:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.158@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7562.152659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.524303] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7574.772940] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7581.159727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7581.169496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7581.184374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7581.687639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7581.700123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7581.706986] Lustre: Skipped 1 previous similar message [ 7581.719098] Lustre: Skipped 1 previous similar message [ 7584.385935] Lustre: server umount lustre-MDT0000 complete [ 7586.789331] LustreError: 173797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7586.817847] LustreError: 173797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7588.956557] LustreError: 172611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236057 with bad export cookie 4621955436826351554 [ 7588.966145] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7588.972882] LustreError: 172611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7589.280277] Lustre: server umount lustre-MDT0001 complete [ 7604.092199] Lustre: server umount lustre-OST0000 complete [ 7612.391599] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 7621.634628] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7659.798965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7660.095572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7660.158544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7660.285556] Lustre: lustre-MDT0000: new disk, initializing [ 7660.387273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7660.401566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7664.715329] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7676.335547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7676.478395] Lustre: 177600:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7676.550966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7676.563564] Lustre: Skipped 1 previous similar message [ 7676.652882] Lustre: lustre-MDT0001: new disk, initializing [ 7676.774607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7676.813443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7676.830118] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7681.244905] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7689.126473] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7693.777831] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7701.195727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7701.554532] Lustre: lustre-OST0000: new disk, initializing [ 7701.560191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7701.576514] Lustre: 179340:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7701.766368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7708.638769] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7711.353058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7711.365576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7711.478600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7716.377868] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7720.906461] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7721.123822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7726.131728] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7726.387750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7732.197281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7732.200649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.203322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7732.205424] LustreError: Skipped 1 previous similar message [ 7732.224336] Lustre: Skipped 1 previous similar message [ 7734.516947] Lustre: server umount lustre-OST0000 complete [ 7742.435329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.457027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.466138] Lustre: Skipped 2 previous similar messages [ 7745.150690] Lustre: server umount lustre-MDT0000 complete [ 7747.553327] LustreError: 177606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7747.575421] LustreError: 177606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7749.077892] LustreError: 177592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236217 with bad export cookie 4621955436826352275 [ 7749.089069] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7749.100483] LustreError: 177592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7749.453743] Lustre: server umount lustre-MDT0001 complete [ 7758.871863] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:50:25 (1781236225) [ 7779.205946] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7782.341561] Key type lgssc unregistered [ 7782.677506] LNet: 182120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7782.693144] LNetError: 182120:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7782.711582] LNet: Removed LNI 192.168.203.158@tcp [ 7783.855277] Key type .llcrypt unregistered [ 7783.857776] Key type ._llcrypt unregistered [ 7803.697586] Key type ._llcrypt registered [ 7803.700394] Key type .llcrypt registered [ 7803.802495] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7804.827802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7804.974706] alg: No test for adler32 (adler32-zlib) [ 7806.114812] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 7806.310449] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 7807.992190] Key type lgssc registered [ 7808.987900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7815.811146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7815.836804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.289803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.723847] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7827.424876] LustreError: 183318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.446195] LustreError: 183318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7828.455647] LustreError: 183319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7830.577454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7830.908608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7835.535703] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7841.112985] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7845.013376] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7852.917962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7853.215450] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7853.221445] LustreError: 184956:0:(llog.c:1687:llog_backup()) MGC192.168.203.158@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7853.230507] Lustre: 184956:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.158@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7853.395631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7860.478866] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7866.649838] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7868.923174] LustreError: 183338:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 7868.944413] LustreError: 183338:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 7871.969521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7871.975654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.996542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7874.025702] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.028206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7874.043179] Lustre: Skipped 1 previous similar message [ 7874.046660] Lustre: Skipped 1 previous similar message [ 7879.140492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7879.151654] Lustre: Skipped 1 previous similar message [ 7883.232448] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7883.507931] Lustre: server umount lustre-MDT0000 complete [ 7884.257321] LustreError: 183318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7884.282222] LustreError: 183318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7887.581396] LustreError: 183298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236356 with bad export cookie 12566766189464543227 [ 7887.594023] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7887.603198] LustreError: 183298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7888.052781] Lustre: server umount lustre-MDT0001 complete [ 7903.053289] Lustre: server umount lustre-OST0000 complete [ 7914.710961] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 7927.198109] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7970.627374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7971.029223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7971.068517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7971.156882] Lustre: lustre-MDT0000: new disk, initializing [ 7971.257880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7971.279930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7975.926335] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7988.792776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.965506] Lustre: 188339:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7989.046820] LustreError: 188351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7989.068884] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7989.073453] Lustre: Skipped 1 previous similar message [ 7989.076902] LustreError: 189106:0:(llog.c:1687:llog_backup()) MGC192.168.203.158@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7989.087668] LustreError: 189106:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7989.092375] Lustre: 189106:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.158@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7989.116606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7989.120486] Lustre: Skipped 1 previous similar message [ 7989.188672] Lustre: lustre-MDT0001: new disk, initializing [ 7989.279544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7989.318267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7989.333601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7994.624972] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8003.149414] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8007.700371] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8015.136404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8015.462561] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8015.466468] Lustre: Skipped 1 previous similar message [ 8015.469948] LustreError: 190058:0:(llog.c:1687:llog_backup()) MGC192.168.203.158@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8015.479957] LustreError: 190058:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8015.486583] Lustre: 190058:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.158@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8015.494383] Lustre: 190058:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8015.518844] Lustre: lustre-OST0000: new disk, initializing [ 8015.524891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8015.533182] Lustre: 190082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8015.617250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8022.318259] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8023.629166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8023.642751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8023.741454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8029.718502] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8034.161430] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8034.402177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8038.786381] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8039.064552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8044.517474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8044.521887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.524559] LustreError: Skipped 1 previous similar message [ 8044.546102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8044.554304] Lustre: Skipped 3 previous similar messages [ 8049.639631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8049.643692] Lustre: Skipped 1 previous similar message [ 8055.263235] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8055.450591] Lustre: server umount lustre-OST0000 complete [ 8059.872596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8059.875216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8059.884218] Lustre: Skipped 2 previous similar messages [ 8059.897228] Lustre: Skipped 2 previous similar messages [ 8067.703986] Lustre: server umount lustre-MDT0000 complete [ 8070.113032] LustreError: 188347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8070.123943] LustreError: 188347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8071.381477] LustreError: 190081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236540 with bad export cookie 12566766189464543948 [ 8071.385594] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8071.394135] LustreError: 190081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8071.728925] Lustre: server umount lustre-MDT0001 complete [ 8082.197471] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:55:47 (1781236547) [ 8103.701493] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8115.797912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8116.373165] LustreError: 193048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8116.430399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8121.477618] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8130.312524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8135.528899] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8139.292861] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8146.660215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8153.918390] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8163.230450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.454724] Lustre: lustre-OST0001: new disk, initializing [ 8163.460796] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8163.471453] Lustre: 195307:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8163.560412] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8163.568339] Lustre: Skipped 2 previous similar messages [ 8170.376894] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8172.643578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8172.662254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8172.745618] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8182.619705] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8190.571661] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8192.415173] Lustre: DEBUG MARKER: ADD OST3 [ 8197.577532] loop: module loaded [ 8197.644082] loop0: detected capacity change from 0 to 204800000 [ 8197.877591] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8197.896230] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8197.905160] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8204.754872] loop0: detected capacity change from 0 to 204800000 [ 8204.904187] Lustre: DEBUG MARKER: STOP OST3 [ 8212.223726] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8212.586479] Lustre: lustre-OST0003: new disk, initializing [ 8212.597952] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8212.610348] Lustre: 197353:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8214.756855] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8214.977867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8214.988533] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8215.047596] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8216.064502] Lustre: Failing over lustre-OST0003 [ 8216.340261] Lustre: server umount lustre-OST0003 complete [ 8216.800759] LustreError: 197506:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8219.809051] LustreError: 195751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.829263] LustreError: 195751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8220.154889] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.160703] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8220.901193] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8233.347426] Lustre: DEBUG MARKER: START OST3 again [ 8241.479873] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8241.913286] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8241.924826] Lustre: Skipped 1 previous similar message [ 8241.947804] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8243.246286] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8243.535815] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8243.535957] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8248.810179] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8269.452414] Lustre: Failing over lustre-OST0003 [ 8269.591711] Lustre: server umount lustre-OST0003 complete [ 8272.887819] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.902384] Lustre: Skipped 1 previous similar message [ 8272.909958] LustreError: 195751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8272.926927] LustreError: 195751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 8289.251748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8289.265526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.292775] Lustre: Skipped 1 previous similar message [ 8289.295650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8289.305576] Lustre: Skipped 2 previous similar messages [ 8295.336977] Lustre: server umount lustre-MDT0000 complete [ 8299.815829] LustreError: 193029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236768 with bad export cookie 12566766189464544830 [ 8299.845295] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8299.853255] LustreError: 193029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8314.335187] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8315.012282] Lustre: server umount lustre-MDT0001 complete [ 8320.369250] Lustre: server umount lustre-OST0000 complete [ 8325.171150] Lustre: server umount lustre-OST0001 complete [ 8333.418849] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 8342.678960] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8386.706497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8387.042265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8387.093093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8387.233114] Lustre: lustre-MDT0000: new disk, initializing [ 8387.333136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8387.374161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8393.018921] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8405.559244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8405.684734] Lustre: 202841:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8405.742121] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8405.747108] Lustre: Skipped 1 previous similar message [ 8405.823925] Lustre: lustre-MDT0001: new disk, initializing [ 8405.981763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8406.001476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8411.116630] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8420.107507] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8424.662511] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8433.200135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8433.497448] Lustre: 204599:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8435.723277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8436.116189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8445.802493] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8457.365070] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8463.535900] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8463.896536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8469.947856] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8470.257477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8476.641290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8476.652631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8476.671407] Lustre: Skipped 5 previous similar messages [ 8476.682528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8476.690435] Lustre: Skipped 12 previous similar messages [ 8478.904429] Lustre: server umount lustre-OST0000 complete [ 8489.767262] Lustre: server umount lustre-MDT0000 complete [ 8492.512086] LustreError: 202849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8492.555951] LustreError: 202849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 8494.153321] LustreError: 203634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236963 with bad export cookie 12566766189464547119 [ 8494.171252] LustreError: 203634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8494.174456] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8494.695645] Lustre: server umount lustre-MDT0001 complete [ 8499.518827] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:02:45 (1781236965) [ 8517.834700] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 8529.553741] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8568.713020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8569.117933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8569.197754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8569.207689] Lustre: Skipped 1 previous similar message [ 8569.327237] Lustre: lustre-MDT0000: new disk, initializing [ 8569.329905] Lustre: Skipped 1 previous similar message [ 8569.445645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8569.455054] Lustre: Skipped 1 previous similar message [ 8574.323515] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8585.146585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8585.260900] Lustre: 209034:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8585.639785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8590.407381] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8598.541957] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8602.139370] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8609.568734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8609.832552] Lustre: 210775:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8611.520452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8616.152986] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8624.390394] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8630.016321] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8630.266478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8636.151658] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8636.563857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8642.027381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8642.044391] LustreError: Skipped 1 previous similar message [ 8642.056650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8642.082756] Lustre: Skipped 3 previous similar messages [ 8642.100973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8642.107769] Lustre: Skipped 4 previous similar messages [ 8645.009784] Lustre: server umount lustre-OST0000 complete [ 8655.668056] Lustre: server umount lustre-MDT0000 complete [ 8657.897914] LustreError: 211969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8657.932250] LustreError: 211969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8659.508045] LustreError: 209026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237128 with bad export cookie 12566766189464548001 [ 8659.511388] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8659.518738] LustreError: 209026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8660.027684] Lustre: server umount lustre-MDT0001 complete [ 8670.011551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.555092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8670.559908] Lustre: Skipped 5 previous similar messages [ 8675.679887] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8684.511429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8689.229107] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8695.436414] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8699.887467] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8706.730128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8713.874489] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8720.314450] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8731.106030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8731.138226] LustreError: Skipped 2 previous similar messages [ 8737.385285] Lustre: server umount lustre-OST0000 complete [ 8748.443158] Lustre: server umount lustre-MDT0000 complete [ 8753.007728] LustreError: 214463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237221 with bad export cookie 12566766189464548890 [ 8753.027129] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8753.031967] LustreError: 214463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8753.504673] Lustre: server umount lustre-MDT0001 complete [ 8859.108730] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 8861.942425] Key type lgssc unregistered [ 8862.249566] LNet: 216151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8862.265448] LNetError: 216151:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8862.281691] LNet: Removed LNI 192.168.203.158@tcp [ 8863.082157] Key type .llcrypt unregistered [ 8863.087832] Key type ._llcrypt unregistered [ 8888.336339] Key type ._llcrypt registered [ 8888.338920] Key type .llcrypt registered [ 8888.446611] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 8904.231304] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8905.225816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8905.388825] alg: No test for adler32 (adler32-zlib) [ 8906.447113] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 8906.653727] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 8908.344659] Key type lgssc registered [ 8909.956611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8946.546912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8946.574237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8947.954659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8948.003516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8948.121894] Lustre: lustre-MDT0000: new disk, initializing [ 8948.229758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8948.254450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8952.561355] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8962.741665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8962.940872] Lustre: 219475:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8963.020202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8963.025263] Lustre: Skipped 1 previous similar message [ 8963.124739] Lustre: lustre-MDT0001: new disk, initializing [ 8963.263739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8963.322535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8963.337272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8967.887569] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8975.035203] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8978.603745] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8986.466668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8986.745848] Lustre: lustre-OST0000: new disk, initializing [ 8986.751817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8986.760840] Lustre: 221217:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8986.858453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8993.568446] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8993.865094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8993.872250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8994.052994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9002.126208] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9007.530518] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9007.754551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9012.900986] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9013.113192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9019.881020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9019.887853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9019.894280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9019.898112] Lustre: Skipped 1 previous similar message [ 9019.925934] Lustre: Skipped 1 previous similar message [ 9021.403644] Lustre: server umount lustre-OST0000 complete [ 9030.113286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9030.118603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.123616] LustreError: Skipped 1 previous similar message [ 9030.125854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9030.152617] Lustre: Skipped 1 previous similar message [ 9031.951541] Lustre: server umount lustre-MDT0000 complete [ 9035.243201] LustreError: 219486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9035.268403] LustreError: 219486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9036.810912] LustreError: 219468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237505 with bad export cookie 4013587311239616862 [ 9036.828263] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9037.152635] Lustre: server umount lustre-MDT0001 complete [ 9059.584797] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 9071.203263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9071.783873] LustreError: 223986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9071.844460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9076.201984] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9077.220589] LustreError: 223987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9081.329301] LustreError: 223986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9085.476981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9085.854867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9090.635900] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9093.614528] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9101.691713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9102.066140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9108.699266] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9117.979176] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9118.132234] Lustre: lustre-OST0001: new disk, initializing [ 9118.135235] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9118.140482] Lustre: 226242:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9118.195886] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9124.589860] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9128.059495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9128.073499] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9128.239743] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9135.704169] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9144.673396] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9158.041332] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:13:44 (1781237624) [ 9162.720482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9162.742956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9162.778886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9162.788853] Lustre: Skipped 1 previous similar message [ 9166.384911] Lustre: server umount lustre-MDT0000 complete [ 9169.381527] LustreError: 223987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9169.396987] LustreError: 223987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9170.199285] LustreError: 223968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237639 with bad export cookie 4013587311239617744 [ 9170.208551] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9170.210788] LustreError: 223968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9170.639140] Lustre: server umount lustre-MDT0001 complete [ 9185.037501] Lustre: server umount lustre-OST0000 complete [ 9199.343808] Lustre: server umount lustre-OST0001 complete [ 9207.125995] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 9217.351145] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 9255.620093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9255.844816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9255.874337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9255.945442] Lustre: lustre-MDT0000: new disk, initializing [ 9256.053136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9256.077278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9260.549286] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9271.845439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9272.047509] Lustre: 230140:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9272.118915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9272.122628] Lustre: Skipped 1 previous similar message [ 9272.240673] Lustre: lustre-MDT0001: new disk, initializing [ 9272.380155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9272.417659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9277.372431] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9286.054258] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9290.212197] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9297.000249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9297.306995] Lustre: lustre-OST0000: new disk, initializing [ 9297.316273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9297.321061] Lustre: 231881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9299.423466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9299.441644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9299.602655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9303.818852] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9312.042145] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9316.843524] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9317.094970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9321.875889] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9322.213180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9325.033059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9325.033586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9325.044619] LustreError: Skipped 1 previous similar message [ 9325.046835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9325.046842] Lustre: Skipped 4 previous similar messages [ 9325.080494] Lustre: Skipped 4 previous similar messages [ 9330.146503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9330.154109] Lustre: Skipped 2 previous similar messages [ 9330.614992] Lustre: server umount lustre-OST0000 complete [ 9335.264668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9340.384491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9340.397607] Lustre: Skipped 3 previous similar messages [ 9341.504958] Lustre: server umount lustre-MDT0000 complete [ 9345.506049] LustreError: 230150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9345.543603] LustreError: 230150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9345.905168] LustreError: 230131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237814 with bad export cookie 4013587311239619879 [ 9345.930791] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9345.932990] LustreError: 230131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9346.324080] Lustre: server umount lustre-MDT0001 complete [ 9355.046411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9355.647197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9355.655536] Lustre: Skipped 2 previous similar messages [ 9360.244761] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9365.988243] LustreError: 233953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9366.016483] LustreError: 233953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9369.708344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9374.991589] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9382.542229] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9386.841287] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9395.950859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9402.016187] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9407.929485] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9554.415550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9554.421671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9554.454892] Lustre: Skipped 1 previous similar message [ 9554.466739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9554.477268] Lustre: Skipped 1 previous similar message [ 9559.835811] Lustre: server umount lustre-OST0000 complete [ 9570.807506] Lustre: server umount lustre-MDT0000 complete [ 9574.582926] LustreError: 233934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238043 with bad export cookie 4013587311239620761 [ 9574.586531] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9574.594198] LustreError: 233934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9575.137170] Lustre: server umount lustre-MDT0001 complete [ 9676.552480] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 9679.493984] Key type lgssc unregistered [ 9679.837080] LNet: 237206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9679.846205] LNetError: 237206:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9679.869916] LNet: Removed LNI 192.168.203.158@tcp [ 9680.689167] Key type .llcrypt unregistered [ 9680.693768] Key type ._llcrypt unregistered [ 9706.666039] Key type ._llcrypt registered [ 9706.668779] Key type .llcrypt registered [ 9706.787803] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 9722.124073] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 9723.423955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9723.515277] alg: No test for adler32 (adler32-zlib) [ 9724.686753] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 9724.915367] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 9726.607660] Key type lgssc registered [ 9727.803298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9773.080308] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 9785.301383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9785.349628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9786.930715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9787.054756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9787.207384] Lustre: lustre-MDT0000: new disk, initializing [ 9787.391687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9787.446920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9792.728736] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9803.461105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9803.657628] Lustre: 241221:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9803.789099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9803.795506] Lustre: Skipped 1 previous similar message [ 9803.930468] Lustre: lustre-MDT0001: new disk, initializing [ 9804.054097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9804.144157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9804.160606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9809.553919] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9815.801335] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9823.562514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9823.882756] Lustre: lustre-OST0000: new disk, initializing [ 9823.885925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9823.892408] Lustre: 242825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9823.978477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9828.462949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9828.481083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9828.637631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9830.102449] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9841.460200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9841.790272] Lustre: lustre-OST0001: new disk, initializing [ 9841.796951] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9841.818545] Lustre: 243679:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9841.999412] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9850.451529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9850.468382] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9850.610470] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9850.796462] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9860.935719] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9867.992697] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9881.057331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9881.062371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9881.072905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9881.584875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9881.593802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9881.601657] Lustre: Skipped 1 previous similar message [ 9886.071654] Lustre: server umount lustre-MDT0000 complete [ 9889.166525] LustreError: 241214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238358 with bad export cookie 6009837143842613758 [ 9889.170349] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9889.173021] LustreError: 241214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9889.361432] Lustre: server umount lustre-MDT0001 complete [ 9904.256817] Lustre: server umount lustre-OST0000 complete [ 9907.168053] Lustre: 238749:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781238360/real 1781238360] req@ffff979470c43c00 x1867763619540480/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781238376 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9907.199530] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.220786] Lustre: Skipped 1 previous similar message [ 9908.313349] Lustre: server umount lustre-OST0001 complete [ 9921.910963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9931.836543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9939.490989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9949.341654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9967.527858] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 9976.002039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9976.023826] Lustre: lustre-MDT0000: reset Object Index mappings [ 9976.293806] LustreError: 248656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9976.306445] LustreError: 248656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9976.363990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9979.645689] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9981.411032] LustreError: 248657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9986.530705] LustreError: 248656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9987.797257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9987.810626] Lustre: lustre-MDT0001: reset Object Index mappings [ 9988.280697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9992.998222] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9995.777737] Lustre: 249761:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10002.505426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10010.716478] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10013.307554] LustreError: 250116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10018.792054] LustreError: 250116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10021.577981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10021.859624] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10021.870934] Lustre: Skipped 1 previous similar message [10022.915216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10022.915819] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10030.177651] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10038.504616] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10041.882162] Lustre: 251600:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10048.489600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10048.492879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10048.511791] Lustre: Skipped 3 previous similar messages [10048.531115] Lustre: Skipped 5 previous similar messages [10049.115412] Lustre: server umount lustre-MDT0000 complete [10053.096240] LustreError: 248636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238521 with bad export cookie 6009837143842620548 [10053.103656] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10053.110282] LustreError: 248636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10053.554739] Lustre: server umount lustre-MDT0001 complete [10068.108930] Lustre: server umount lustre-OST0000 complete [10070.048472] Lustre: 238751:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781238522/real 1781238522] req@ffff979346b67c00 x1867763619589760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781238538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10070.083086] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10072.126355] Lustre: server umount lustre-OST0001 complete [10091.195718] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:29:17 (1781238557) [10108.101266] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [10120.392254] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [10157.437029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10157.619882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10157.658748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10157.737210] Lustre: lustre-MDT0000: new disk, initializing [10157.817348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10157.830572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10161.432375] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10171.872683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10172.036700] Lustre: 255819:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10172.120806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10172.132723] Lustre: Skipped 1 previous similar message [10172.219332] Lustre: lustre-MDT0001: new disk, initializing [10172.403510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10172.411960] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10176.833568] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10184.313175] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10188.433222] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10196.334427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10196.678790] Lustre: lustre-OST0000: new disk, initializing [10196.685453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10196.692334] Lustre: 257563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10198.582462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10198.596344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10198.716675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10203.035279] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10211.195549] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10216.431111] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10216.696805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10221.459765] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10221.679911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10224.612421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10224.615336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10224.617831] LustreError: Skipped 1 previous similar message [10224.625230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10224.646649] Lustre: Skipped 1 previous similar message [10224.679081] Lustre: Skipped 1 previous similar message [10229.733607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10229.738047] Lustre: Skipped 1 previous similar message [10230.075344] Lustre: server umount lustre-OST0000 complete [10239.971639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10239.985789] Lustre: Skipped 3 previous similar messages [10240.384348] Lustre: server umount lustre-MDT0000 complete [10244.664865] LustreError: 257575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238713 with bad export cookie 6009837143842622641 [10244.687501] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10244.980317] Lustre: server umount lustre-MDT0001 complete [10263.369770] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [10273.085924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10273.426457] LustreError: 260336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10273.499815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10273.509817] Lustre: Skipped 2 previous similar messages [10277.939964] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10287.220093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.346840] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10295.456070] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10302.078527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10308.767946] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10318.155802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10318.380722] Lustre: lustre-OST0001: new disk, initializing [10318.390173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10318.401375] Lustre: 262592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10320.356768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10320.374658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10320.520880] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10325.196090] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10334.628604] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10338.322474] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10343.250620] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10343.352516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10351.078656] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10351.088985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10351.091170] Lustre: Skipped 4 previous similar messages [10351.097023] Lustre: Skipped 1 previous similar message [10364.383126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10364.561901] Lustre: server umount lustre-MDT0000 complete [10366.442225] LustreError: 261327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10366.465674] LustreError: 261327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10368.321129] LustreError: 260316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238837 with bad export cookie 6009837143842623523 [10368.322591] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10368.341030] LustreError: 260316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10368.713257] Lustre: server umount lustre-MDT0001 complete [10382.348061] Lustre: server umount lustre-OST0000 complete [10396.851737] Lustre: server umount lustre-OST0001 complete [10415.116257] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [10427.874747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10428.482889] LustreError: 265633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10428.597343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10428.601395] Lustre: Skipped 3 previous similar messages [10433.314277] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10441.761359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10446.394683] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10449.748825] Lustre: 266739:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10456.929662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10463.584633] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10472.972188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10474.383500] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10474.459155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10480.230965] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10487.939675] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10491.533797] Lustre: 268582:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10496.847322] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10496.946052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10506.913717] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:36:13 (1781238973) [10524.135364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10524.145272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10524.156201] Lustre: Skipped 1 previous similar message [10524.161363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10524.166019] Lustre: Skipped 9 previous similar messages [10527.827825] Lustre: server umount lustre-MDT0000 complete [10530.786368] LustreError: 267114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.811258] LustreError: 267114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10531.251276] LustreError: 265615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239000 with bad export cookie 6009837143842630061 [10531.259786] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10531.262052] LustreError: 265615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10537.624837] Lustre: server umount lustre-MDT0001 complete [10547.492940] Lustre: server umount lustre-OST0000 complete [10550.927153] Lustre: server umount lustre-OST0001 complete [10557.096941] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [10566.934286] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [10613.995793] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [10624.088994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10624.352332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10624.408582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10624.495266] Lustre: lustre-MDT0000: new disk, initializing [10624.593614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10628.866608] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10638.656260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10638.758199] Lustre: 272768:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10638.990803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10642.682424] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10653.063978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10653.343314] Lustre: 274369:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10655.222613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10658.219524] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10666.791125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10666.943732] Lustre: 275223:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10669.176696] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10672.420635] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10681.116036] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10684.389916] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10684.398396] Lustre: Skipped 1 previous similar message [10698.084920] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:39:24 (1781239164) [10700.258557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10700.266072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10700.285597] Lustre: Skipped 5 previous similar messages [10700.292246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10700.296411] Lustre: Skipped 8 previous similar messages [10702.377160] Lustre: server umount lustre-MDT0000 complete [10704.867604] LustreError: 273638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10704.890976] LustreError: 273638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10705.689988] LustreError: 272763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239174 with bad export cookie 6009837143842637698 [10705.691356] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10705.699421] LustreError: 272763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10706.026963] Lustre: server umount lustre-MDT0001 complete [10719.347658] Lustre: server umount lustre-OST0000 complete [10733.053969] Lustre: server umount lustre-OST0001 complete [10743.486232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10744.432701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10744.450794] Lustre: Skipped 7 previous similar messages [10750.255747] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10760.670771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10765.878692] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10771.706170] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10774.916981] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10781.619411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10788.699143] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10794.503391] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10811.647250] loop1: detected capacity change from 0 to 204800000 [10811.939518] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10811.960442] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10811.976207] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10817.280776] loop1: detected capacity change from 0 to 204800000 [10822.954459] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10823.080874] LustreError: 277438:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs19482: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10823.105245] LustreError: 277438:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs19482 [10823.114847] LustreError: 277438:0:(mgs_handler.c:517:mgs_target_reg()) fs19482-MDT0001: Error checking target index: rc = -13 [10823.130336] LustreError: fs19482-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10823.138451] LustreError: 280827:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10823.143786] LustreError: 280827:0:(tgt_mount.c:2084:server_put_super()) no obd fs19482-MDT0001 [10823.148819] LustreError: 280827:0:(tgt_mount.c:116:server_deregister_mount()) fs19482-MDT0001 not registered [10823.191584] Lustre: server umount fs19482-MDT0001 complete [10823.198559] LustreError: 280827:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10832.316370] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10832.608806] Lustre: Setting parameter fs19482-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs19482-MDT0001 [10832.701445] Lustre: srv-fs19482-MDT0001: No data found on store. Initialize space. [10832.707785] Lustre: Skipped 4 previous similar messages [10832.833411] Lustre: fs19482-MDT0001: new disk, initializing [10832.838613] Lustre: Skipped 3 previous similar messages [10832.966756] Lustre: srv-fs19482-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10832.980652] Lustre: 281294:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19482-MDT0001-mdtlov: can't create new llog: rc = -115 [10832.995951] LustreError: 281294:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19482-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10837.604659] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10845.078732] Lustre: server umount fs19482-MDT0001 complete [10858.574502] loop1: detected capacity change from 0 to 204800000 [10864.307249] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10864.606916] Lustre: 282847:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19482-MDT0001-mdtlov: can't create new llog: rc = -115 [10864.612508] LustreError: 282847:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19482-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10868.625770] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10871.271098] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10873.647920] Lustre: server umount fs19482-MDT0001 complete [10884.066029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10884.070537] LustreError: Skipped 1 previous similar message [10884.076751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10884.096874] Lustre: Skipped 3 previous similar messages [10884.103552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10889.807775] Lustre: server umount lustre-MDT0000 complete [10899.117226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10899.349973] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10904.520769] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10905.087766] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10913.186254] loop1: detected capacity change from 0 to 204800000 [10918.917297] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10919.327841] Lustre: 285262:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19482-MDT0001-mdtlov: can't create new llog: rc = -115 [10919.344289] LustreError: 285262:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19482-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10923.495640] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10928.398906] Lustre: server umount fs19482-MDT0001 complete [10955.243249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10959.688295] Lustre: server umount lustre-OST0000 complete [10971.618848] LustreError: 277447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10971.649915] LustreError: 277447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 110 previous similar messages [10972.756963] Lustre: server umount lustre-MDT0000 complete [10977.283377] LustreError: 277430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239446 with bad export cookie 6009837143842640379 [10977.294414] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10977.300841] LustreError: 277430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10977.842866] Lustre: server umount lustre-MDT0001 complete [10991.922978] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [10995.113446] Key type lgssc unregistered [10995.474518] LNet: 287572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10995.492184] LNetError: 287572:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10996.522649] LNet: Removed LNI 192.168.203.158@tcp [10997.351267] Key type .llcrypt unregistered [10997.359644] Key type ._llcrypt unregistered [11020.506504] Key type ._llcrypt registered [11020.509504] Key type .llcrypt registered [11020.621898] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [11022.104281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11022.275690] alg: No test for adler32 (adler32-zlib) [11023.513200] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [11023.855537] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [11025.631379] Key type lgssc registered [11027.026232] Lustre: Echo OBD driver; http://www.lustre.org/ [11034.863400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11034.892290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11036.555906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11040.708862] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11046.885501] LustreError: 288830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11046.906288] LustreError: 288830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11049.704406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.242910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11054.545399] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11060.693686] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11064.634868] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11072.379317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11072.919559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11076.009549] LustreError: 290432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11079.681379] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11081.193030] LustreError: 290434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11085.619407] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11087.337501] LustreError: 290433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11087.364494] LustreError: 290433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11092.451648] LustreError: 290854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11092.490734] LustreError: 290854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11102.707981] LustreError: 290433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.725029] LustreError: 290433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11104.709710] loop1: detected capacity change from 0 to 204800000 [11104.961264] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11104.970597] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11104.982697] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11110.403182] loop1: detected capacity change from 0 to 204800000 [11115.947175] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11116.105382] Lustre: fs2034-OST0000: new disk, initializing [11116.108722] Lustre: srv-fs2034-OST0000: No data found on store. Initialize space. [11116.114245] Lustre: 292223:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11116.173841] Lustre: fs2034-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11119.235589] LustreError: 290432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11119.260279] LustreError: 290432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [11122.321109] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11129.958153] Lustre: server umount fs2034-OST0000 complete [11142.117225] loop1: detected capacity change from 0 to 204800000 [11147.534288] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11147.740416] Lustre: fs2034-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11153.929822] LustreError: 290854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11153.972135] LustreError: 290854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [11154.640391] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11159.019180] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11164.136958] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11165.421529] loop2: detected capacity change from 0 to 204800000 [11165.600558] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11165.654000] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11165.670723] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11171.919295] loop2: detected capacity change from 0 to 204800000 [11178.525367] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11178.570395] LustreError: fs2034-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11178.576843] LustreError: fs2034-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11178.593686] LustreError: 295231:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11178.604350] LustreError: 295231:0:(tgt_mount.c:2084:server_put_super()) no obd fs2034-OST0001 [11178.611871] LustreError: 295231:0:(tgt_mount.c:116:server_deregister_mount()) fs2034-OST0001 not registered [11178.646167] Lustre: server umount fs2034-OST0001 complete [11178.651551] LustreError: 295231:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11189.478127] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11189.693710] Lustre: fs2034-OST0001: new disk, initializing [11189.700634] Lustre: srv-fs2034-OST0001: No data found on store. Initialize space. [11189.719259] Lustre: 295784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11189.840053] Lustre: fs2034-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11196.800250] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11206.405192] Lustre: server umount fs2034-OST0000 complete [11216.660792] Lustre: server umount fs2034-OST0001 complete [11220.453810] LustreError: 290434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11220.483304] LustreError: 290434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11230.734365] loop1: detected capacity change from 0 to 204800000 [11236.952540] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11237.243947] Lustre: fs2034-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11244.195581] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11252.214959] loop2: detected capacity change from 0 to 204800000 [11258.262104] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11264.990958] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11273.956933] Lustre: server umount fs2034-OST0000 complete [11283.769455] Lustre: server umount fs2034-OST0001 complete [11300.333899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11300.349885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11300.364530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11302.370882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11302.377092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11302.400321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11307.500760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11307.511617] Lustre: Skipped 2 previous similar messages [11312.615429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11312.625948] Lustre: Skipped 1 previous similar message [11312.628219] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11313.119260] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11313.285627] Lustre: server umount lustre-OST0000 complete [11317.735772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11317.757340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11323.861729] Lustre: server umount lustre-MDT0000 complete [11328.458349] LustreError: 288810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239797 with bad export cookie 15806732721715899024 [11328.484734] LustreError: 288810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11328.485816] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11328.901922] Lustre: server umount lustre-MDT0001 complete [11340.674747] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [11343.558278] Key type lgssc unregistered [11343.802939] LNet: 301643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11343.813104] LNetError: 301643:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11343.832146] LNet: Removed LNI 192.168.203.158@tcp [11344.878133] Key type .llcrypt unregistered [11344.881774] Key type ._llcrypt unregistered [11379.561623] Key type ._llcrypt registered [11379.564754] Key type .llcrypt registered [11379.669395] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [11398.544228] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [11399.851607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11400.089674] alg: No test for adler32 (adler32-zlib) [11401.284896] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [11401.719143] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [11403.551258] Key type lgssc registered [11405.247929] Lustre: Echo OBD driver; http://www.lustre.org/ [11448.063543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11448.096953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11449.470638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11449.559432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11449.688210] Lustre: lustre-MDT0000: new disk, initializing [11449.834242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11449.855966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11455.321154] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11467.690138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11467.809549] Lustre: 305469:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11467.925521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11467.933356] Lustre: Skipped 1 previous similar message [11468.085354] Lustre: lustre-MDT0001: new disk, initializing [11468.209632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11468.276243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11468.288362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11473.322580] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11481.726327] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11486.147183] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11493.634488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11493.953670] Lustre: lustre-OST0000: new disk, initializing [11493.956942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11493.963883] Lustre: 307212:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11494.050852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11500.639922] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11503.708143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11503.726045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11503.867926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11508.971040] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11511.569052] Lustre: Mounted lustre-client [11512.548295] Lustre: Unmounted lustre-client [11524.580713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11524.583985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11524.605799] LustreError: Skipped 1 previous similar message [11524.612425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11524.632537] Lustre: Skipped 1 previous similar message [11524.651779] Lustre: Skipped 1 previous similar message [11527.155636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11527.168395] Lustre: Skipped 1 previous similar message [11529.484762] Lustre: server umount lustre-OST0000 complete [11534.825121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11534.840271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11534.869743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11537.382950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11537.383115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.582292] Lustre: server umount lustre-MDT0000 complete [11543.028001] LustreError: 305462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240011 with bad export cookie 16851985242009824556 [11543.030436] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11543.037834] LustreError: 305462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11543.436462] Lustre: server umount lustre-MDT0001 complete [11555.160267] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [11558.725203] Key type lgssc unregistered [11570.719147] LNet: 1 local NIs in recovery (showing 1): 192.168.203.158@tcp [11571.362058] LNet: 309241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11571.372347] LNetError: 309241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11642.087600] LNet: Removed LNI 192.168.203.158@tcp [11643.029890] Key type .llcrypt unregistered [11643.032092] Key type ._llcrypt unregistered [11663.861353] Key type ._llcrypt registered [11663.870401] Key type .llcrypt registered [11663.998466] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [11680.400814] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [11681.494557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11681.703795] alg: No test for adler32 (adler32-zlib) [11682.813441] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [11683.049046] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [11684.736527] Key type lgssc registered [11686.038678] Lustre: Echo OBD driver; http://www.lustre.org/ [11718.783754] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:56:24 (1781240184) [11725.450575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11725.465179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11726.729459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11726.753628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11726.814853] Lustre: lustre-MDT0000: new disk, initializing [11726.891590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11726.911733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11731.705207] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11743.319246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11743.484869] Lustre: 312426:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11743.560659] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11743.563086] Lustre: Skipped 1 previous similar message [11743.714898] Lustre: lustre-MDT0001: new disk, initializing [11743.830315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11743.869843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11743.886049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11748.054889] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11756.607776] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11760.519091] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11767.660170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11767.952194] Lustre: lustre-OST0000: new disk, initializing [11767.956703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11767.964605] Lustre: 314170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11768.072766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11773.529463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11773.553310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11773.688078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11774.563407] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11783.614951] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11794.408962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11794.412962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.413492] LustreError: Skipped 1 previous similar message [11794.419673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11794.428791] Lustre: Skipped 1 previous similar message [11794.447498] Lustre: Skipped 1 previous similar message [11799.525652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11799.534473] Lustre: Skipped 1 previous similar message [11804.646912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11804.658222] Lustre: Skipped 1 previous similar message [11808.223342] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11808.519103] Lustre: server umount lustre-OST0000 complete [11814.884474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11814.888486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11814.899658] Lustre: Skipped 1 previous similar message [11814.917967] Lustre: Skipped 1 previous similar message [11819.852100] Lustre: server umount lustre-MDT0000 complete [11823.952305] LustreError: 314169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240292 with bad export cookie 4873683259464257825 [11823.955390] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11823.962107] LustreError: 314169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11824.389357] Lustre: server umount lustre-MDT0001 complete [11836.098876] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [11839.328103] Key type lgssc unregistered [11839.524716] LNet: 315803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11839.534454] LNetError: 315803:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11839.548843] LNet: Removed LNI 192.168.203.158@tcp [11840.422168] Key type .llcrypt unregistered [11840.425163] Key type ._llcrypt unregistered [11869.466401] Key type ._llcrypt registered [11869.470852] Key type .llcrypt registered [11869.609665] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [11870.823021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11870.867934] alg: No test for adler32 (adler32-zlib) [11872.135385] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [11872.351414] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [11874.071220] Key type lgssc registered [11875.130414] Lustre: Echo OBD driver; http://www.lustre.org/ [11884.117966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11884.133423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11885.774039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11890.405545] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11895.332133] LustreError: 317210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.350025] LustreError: 317210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11899.521092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11899.920192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11904.554732] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11911.652696] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11916.953294] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11924.287912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11924.385467] LustreError: 318802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11924.594561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11931.084953] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11936.757568] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11945.416168] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11947.114197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11952.101470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11952.112603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11952.120811] Lustre: Skipped 1 previous similar message [11954.846375] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.58@tcp (stopping) [11957.232284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11957.235484] Lustre: Skipped 2 previous similar messages [11957.476482] Lustre: server umount lustre-MDT0000 complete [11959.933465] LustreError: 317205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11961.314349] LustreError: 317191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240430 with bad export cookie 14658608884772847750 [11961.332604] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11961.334093] LustreError: 317191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11962.341078] LustreError: 318422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11962.343450] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11962.377651] Lustre: Skipped 2 previous similar messages [11962.382711] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11966.370363] LustreError: 317204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11966.392402] LustreError: 317204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11967.805559] Lustre: server umount lustre-MDT0001 complete [11972.238800] Lustre: server umount lustre-OST0000 complete [11981.943300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11986.814153] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11989.703536] Lustre: 320934:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.158@tcp [11992.194175] Lustre: server umount MGS complete [12001.937390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12002.615433] LustreError: 321518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12002.803307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12008.066882] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12017.462870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12017.826948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12023.114782] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12032.771506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12033.105803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12039.982656] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12047.051429] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12049.368602] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12054.495278] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12056.654668] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12068.325489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12068.331808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12068.337949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12068.343893] Lustre: Skipped 1 previous similar message [12068.350233] Lustre: Skipped 2 previous similar messages [12069.548775] Lustre: server umount lustre-OST0000 complete [12073.446973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12078.563293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12078.567344] Lustre: Skipped 3 previous similar messages [12079.015879] Lustre: server umount lustre-MDT0000 complete [12082.630195] LustreError: 321500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240551 with bad export cookie 14658608884772848597 [12082.638318] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12082.851683] Lustre: server umount lustre-MDT0001 complete [12095.439667] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12098.734451] Key type lgssc unregistered [12099.170526] LNet: 324695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12099.179461] LNetError: 324695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12099.206334] LNet: Removed LNI 192.168.203.158@tcp [12100.284584] Key type .llcrypt unregistered [12100.288702] Key type ._llcrypt unregistered [12121.298396] Key type ._llcrypt registered [12121.302982] Key type .llcrypt registered [12121.447953] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [12137.644472] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [12138.810274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12138.896284] alg: No test for adler32 (adler32-zlib) [12140.020252] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [12140.329351] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [12142.072521] Key type lgssc registered [12143.495965] Lustre: Echo OBD driver; http://www.lustre.org/ [12185.995900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12186.031121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12187.660433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12187.767606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12187.987151] Lustre: lustre-MDT0000: new disk, initializing [12188.167076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12188.283572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12193.073882] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12208.283751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12208.411658] Lustre: 327830:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12208.455978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12208.465056] Lustre: Skipped 1 previous similar message [12208.553696] Lustre: lustre-MDT0001: new disk, initializing [12208.716294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12208.755133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12208.773067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12214.122058] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12223.471756] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12227.569647] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12236.036963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12236.448096] Lustre: lustre-OST0000: new disk, initializing [12236.456165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12236.468053] Lustre: 329577:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12236.653303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12243.252106] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12246.140122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12246.154402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12246.218501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12250.734610] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12255.620216] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12255.822626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12260.759314] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12260.984740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12266.979823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12266.983854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12267.000103] Lustre: Skipped 1 previous similar message [12267.008179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12267.018583] Lustre: Skipped 1 previous similar message [12269.343429] Lustre: server umount lustre-OST0000 complete [12275.183688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12275.205383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12275.219577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12277.220479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12277.221051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12280.015867] Lustre: server umount lustre-MDT0000 complete [12284.503098] LustreError: 327824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240753 with bad export cookie 5035137051731425864 [12284.513535] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12284.520466] LustreError: 327824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12285.140273] Lustre: server umount lustre-MDT0001 complete [12295.047123] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:06:01 (1781240761) [12313.082530] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [12324.073680] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [12364.117836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12364.492547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12364.534220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12364.624446] Lustre: lustre-MDT0000: new disk, initializing [12364.740501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12364.774853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12369.520710] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12381.631597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12381.770720] Lustre: 334153:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12381.885426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12381.902049] Lustre: Skipped 1 previous similar message [12382.066448] Lustre: lustre-MDT0001: new disk, initializing [12382.271907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12382.311661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12382.333860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12387.606869] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12396.333229] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12400.422372] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12407.952809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12408.379624] Lustre: lustre-OST0000: new disk, initializing [12408.384351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12408.398075] Lustre: 335900:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12408.504097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12414.862546] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12417.089785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12417.098336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12417.192079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12422.538519] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12463.584025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12463.593945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12463.598290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12463.618712] Lustre: Skipped 1 previous similar message [12463.649219] Lustre: Skipped 2 previous similar messages [12468.709062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12468.718501] Lustre: Skipped 1 previous similar message [12474.335400] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12474.638931] Lustre: server umount lustre-OST0000 complete [12478.945205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12478.969838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12478.975804] Lustre: Skipped 3 previous similar messages [12493.280671] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12493.464888] LustreError: 337212:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000b4790286) refcount nonzero (1) after lock cleanup; forcing cleanup. [12493.667258] Lustre: server umount lustre-MDT0000 complete [12494.304328] LustreError: 334167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12494.342721] LustreError: 334167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12498.394516] LustreError: 334146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240967 with bad export cookie 5035137051731426746 [12498.408737] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12498.889822] Lustre: server umount lustre-MDT0001 complete [12511.150654] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12514.807994] Key type lgssc unregistered [12515.130835] LNet: 337948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12515.143057] LNetError: 337948:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12516.200321] LNet: Removed LNI 192.168.203.158@tcp [12517.395624] Key type .llcrypt unregistered [12517.398706] Key type ._llcrypt unregistered [12622.307981] Key type ._llcrypt registered [12622.310667] Key type .llcrypt registered [12622.420952] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12622.695416] Key type .llcrypt unregistered [12622.697904] Key type ._llcrypt unregistered [12714.523801] Key type ._llcrypt registered [12714.528704] Key type .llcrypt registered [12714.762682] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12715.287646] Key type .llcrypt unregistered [12715.297277] Key type ._llcrypt unregistered [12749.684287] Key type ._llcrypt registered [12749.688398] Key type .llcrypt registered [12749.842457] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12750.164180] Key type .llcrypt unregistered [12750.169187] Key type ._llcrypt unregistered [12785.349223] Key type ._llcrypt registered [12785.352370] Key type .llcrypt registered [12785.462360] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12785.717408] Key type .llcrypt unregistered [12785.720513] Key type ._llcrypt unregistered [12868.695950] Key type ._llcrypt registered [12868.699632] Key type .llcrypt registered [12868.827333] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12869.121426] Key type .llcrypt unregistered [12869.124361] Key type ._llcrypt unregistered [12903.186196] Key type ._llcrypt registered [12903.187916] Key type .llcrypt registered [12903.307027] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12903.675685] Key type .llcrypt unregistered [12903.679140] Key type ._llcrypt unregistered [12939.984501] Key type ._llcrypt registered [12939.995631] Key type .llcrypt registered [12940.160502] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12940.486601] Key type .llcrypt unregistered [12940.488965] Key type ._llcrypt unregistered [12968.263734] Key type ._llcrypt registered [12968.268862] Key type .llcrypt registered [12968.382305] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12968.694339] Key type .llcrypt unregistered [12968.697314] Key type ._llcrypt unregistered [12998.587501] Key type ._llcrypt registered [12998.591772] Key type .llcrypt registered [12998.725977] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12999.111432] Key type .llcrypt unregistered [12999.117153] Key type ._llcrypt unregistered [13030.748442] Key type ._llcrypt registered [13030.750080] Key type .llcrypt registered [13030.882568] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [13031.179427] Key type .llcrypt unregistered [13031.181094] Key type ._llcrypt unregistered [13093.471399] Key type ._llcrypt registered [13093.474195] Key type .llcrypt registered [13093.565909] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [13093.869414] Key type .llcrypt unregistered [13093.872868] Key type ._llcrypt unregistered [13124.437716] Key type ._llcrypt registered [13124.441298] Key type .llcrypt registered [13124.536377] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [13124.757275] Key type .llcrypt unregistered [13124.758926] Key type ._llcrypt unregistered [13157.534659] Key type ._llcrypt registered [13157.536948] Key type .llcrypt registered [13157.654160] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [13157.932882] Key type .llcrypt unregistered [13157.934739] Key type ._llcrypt unregistered [13210.961381] Key type ._llcrypt registered [13210.963625] Key type .llcrypt registered [13211.071129] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [13231.091706] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [13232.265201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13232.531654] alg: No test for adler32 (adler32-zlib) [13233.844801] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [13234.099700] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [13235.775152] Key type lgssc registered [13237.192661] Lustre: Echo OBD driver; http://www.lustre.org/ [13278.957740] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13278.980497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13280.198372] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13280.264113] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13280.365340] Lustre: tst250fs-MDT0000: new disk, initializing [13280.458841] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13280.479403] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13285.035478] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13297.626580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13297.782899] Lustre: 358012:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13297.853812] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13297.858750] Lustre: Skipped 1 previous similar message [13297.918531] Lustre: tst250fs-MDT0001: new disk, initializing [13298.026945] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13298.072214] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13298.082989] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13302.552882] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13307.540918] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13314.351947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13314.606895] Lustre: tst250fs-OST0000: new disk, initializing [13314.611636] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13314.616742] Lustre: 359610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13314.705692] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13319.735550] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13319.746400] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13319.848641] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13320.833602] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13330.946568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13331.127853] Lustre: tst250fs-OST0001: new disk, initializing [13331.138026] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13331.157577] Lustre: 360466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13331.285611] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13337.283476] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13339.245604] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13339.254562] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13339.345918] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13343.488353] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13345.917937] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13345.929482] Lustre: Skipped 1 previous similar message [13369.826287] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13369.844233] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13369.882082] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13370.353817] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13370.363330] Lustre: Skipped 1 previous similar message [13373.779559] Lustre: server umount tst250fs-MDT0000 complete [13377.543358] LustreError: 358003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241846 with bad export cookie 17239746183018288558 [13377.551235] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13377.554833] LustreError: 358003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13377.894757] Lustre: server umount tst250fs-MDT0001 complete [13392.205573] Lustre: server umount tst250fs-OST0000 complete [13405.343141] Lustre: server umount tst250fs-OST0001 complete [13425.723668] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [13433.666734] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [13463.635643] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:25:30 (1781241930) [13465.330794] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13482.206676] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [13492.971563] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [13528.714589] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13275 sec ========= 01:26:35 (1781241995) [13530.211076] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:26:36 (1781241996) === [13533.337233] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:26:39 (1781241999) === [13571.053671] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [13573.960408] Key type lgssc unregistered [13574.250813] LNet: 369510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13574.262326] LNetError: 369510:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13574.284453] LNet: Removed LNI 192.168.203.158@tcp [13575.168165] Key type .llcrypt unregistered [13575.172137] Key type ._llcrypt unregistered