[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485500194 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003114] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008529] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010021] pid_max: default: 32768 minimum: 301 [ 0.011174] LSM: Security Framework initializing [ 0.012077] Yama: becoming mindful. [ 0.013053] SELinux: Initializing. [ 0.014094] *** VALIDATE selinux *** [ 0.022796] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027464] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028184] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031144] *** VALIDATE tmpfs *** [ 0.033182] *** VALIDATE proc *** [ 0.034317] *** VALIDATE cgroup *** [ 0.035015] *** VALIDATE cgroup2 *** [ 0.036283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039042] Spectre V2 : User space: Vulnerable [ 0.040016] Speculative Store Bypass: Vulnerable [ 0.043536] debug: unmapping init [mem 0xffffffffa6659000-0xffffffffa6660fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046754] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047032] ... version: 2 [ 0.048019] ... bit width: 48 [ 0.049018] ... generic registers: 4 [ 0.050019] ... value mask: 0000ffffffffffff [ 0.051023] ... max period: 00007fffffffffff [ 0.052024] ... fixed-purpose events: 3 [ 0.053019] ... event mask: 000000070000000f [ 0.055282] rcu: Hierarchical SRCU implementation. [ 0.057779] smp: Bringing up secondary CPUs ... [ 0.058711] x86: Booting SMP configuration: [ 0.059035] .... node #0, CPUs: #1 #2 #3 [ 0.062677] smp: Brought up 1 node, 4 CPUs [ 0.064018] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.257344] node 0 deferred pages initialised in 190ms [ 0.261078] devtmpfs: initialized [ 0.262277] x86/mm: Memory block size: 128MB [ 0.265886] gcov: version magic: 0x41383552 [ 0.269323] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.273094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.275331] pinctrl core: initialized pinctrl subsystem [ 0.278186] [ 0.278786] ************************************************************* [ 0.281017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.283014] ** ** [ 0.286015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.289017] ** ** [ 0.291018] ** This means that this kernel is built to expose internal ** [ 0.294016] ** IOMMU data structures, which may compromise security on ** [ 0.296013] ** your system. ** [ 0.299015] ** ** [ 0.302016] ** If you see this message and you are not debugging the ** [ 0.304015] ** kernel, report this immediately to your vendor! ** [ 0.306012] ** ** [ 0.309016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.311013] ************************************************************* [ 0.314629] NET: Registered protocol family 16 [ 0.316455] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.319063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.322070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.326147] cpuidle: using governor menu [ 0.328000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.331488] PCI: Using configuration type 1 for base access [ 0.333134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.342122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.345026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.349065] cryptd: max_cpu_qlen set to 1000 [ 0.351300] ACPI: Added _OSI(Module Device) [ 0.353021] ACPI: Added _OSI(Processor Device) [ 0.355017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.357018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.362168] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.371533] ACPI: Interpreter enabled [ 0.373069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.375018] ACPI: Using IOAPIC for interrupt routing [ 0.377143] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.381433] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.392077] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.395050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.398023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.401095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.407353] acpiphp: Slot [2] registered [ 0.408115] acpiphp: Slot [5] registered [ 0.410128] acpiphp: Slot [6] registered [ 0.412152] acpiphp: Slot [7] registered [ 0.413120] acpiphp: Slot [8] registered [ 0.415147] acpiphp: Slot [9] registered [ 0.417127] acpiphp: Slot [10] registered [ 0.419129] acpiphp: Slot [3] registered [ 0.420131] acpiphp: Slot [4] registered [ 0.422113] acpiphp: Slot [11] registered [ 0.424140] acpiphp: Slot [12] registered [ 0.425124] acpiphp: Slot [13] registered [ 0.427123] acpiphp: Slot [14] registered [ 0.428086] acpiphp: Slot [15] registered [ 0.430106] acpiphp: Slot [16] registered [ 0.431101] acpiphp: Slot [17] registered [ 0.433113] acpiphp: Slot [18] registered [ 0.434110] acpiphp: Slot [19] registered [ 0.437153] acpiphp: Slot [20] registered [ 0.439127] acpiphp: Slot [21] registered [ 0.441108] acpiphp: Slot [22] registered [ 0.443138] acpiphp: Slot [23] registered [ 0.444121] acpiphp: Slot [24] registered [ 0.446105] acpiphp: Slot [25] registered [ 0.448132] acpiphp: Slot [26] registered [ 0.449175] acpiphp: Slot [27] registered [ 0.451111] acpiphp: Slot [28] registered [ 0.453103] acpiphp: Slot [29] registered [ 0.454105] acpiphp: Slot [30] registered [ 0.456106] acpiphp: Slot [31] registered [ 0.457074] PCI host bridge to bus 0000:00 [ 0.459029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.462037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.465031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.469029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.472026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.475028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.477237] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.480087] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.484525] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.497016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.501970] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.504020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.507022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.510021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.513483] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.517017] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.520060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.522846] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.529017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.542951] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.547015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.553467] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.566018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.583017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.606014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.616411] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.625016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.632017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.654021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.667110] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.679020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.686014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.708019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.718681] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.724014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.731017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.747017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.757451] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.762020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.768016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.784032] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.794503] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.802988] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.810027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.829024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.843768] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.845337] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.847338] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.849318] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.851202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.856167] iommu: Default domain type: Passthrough [ 0.858536] SCSI subsystem initialized [ 0.860287] ACPI: bus type USB registered [ 0.862157] usbcore: registered new interface driver usbfs [ 0.864091] usbcore: registered new interface driver hub [ 0.866320] usbcore: registered new device driver usb [ 0.868303] pps_core: LinuxPPS API ver. 1 registered [ 0.871018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.875085] PTP clock support registered [ 0.878263] EDAC MC: Ver: 3.0.0 [ 0.880005] PCI: Using ACPI for IRQ routing [ 0.882839] NetLabel: Initializing [ 0.884018] NetLabel: domain hash size = 128 [ 0.886016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.889114] NetLabel: unlabeled traffic allowed by default [ 0.891389] vgaarb: loaded [ 0.893303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.895013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.903000] clocksource: Switched to clocksource kvm-clock [ 1.015985] VFS: Disk quotas dquot_6.6.0 [ 1.018058] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.022418] *** VALIDATE ramfs *** [ 1.023735] *** VALIDATE hugetlbfs *** [ 1.025682] pnp: PnP ACPI init [ 1.028576] pnp: PnP ACPI: found 6 devices [ 1.045738] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.050062] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.052660] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.055279] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.058176] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.061130] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.064361] NET: Registered protocol family 2 [ 1.066982] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.072096] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.075417] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.080627] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.084238] TCP: Hash tables configured (established 65536 bind 65536) [ 1.087822] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.091467] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.094902] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.098413] NET: Registered protocol family 1 [ 1.102153] RPC: Registered named UNIX socket transport module. [ 1.104817] RPC: Registered udp transport module. [ 1.106491] RPC: Registered tcp transport module. [ 1.108665] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.111823] NET: Registered protocol family 44 [ 1.113621] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.116118] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.118313] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.120962] PCI: CLS 0 bytes, default 64 [ 1.122818] Unpacking initramfs... [ 2.565259] debug: unmapping init [mem 0xffff8b6f7cc54000-0xffff8b6f7ffbffff] [ 2.569721] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.572417] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.576833] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.087663] Initialise system trusted keyrings [ 3.089706] Key type blacklist registered [ 3.092247] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.113604] zbud: loaded [ 3.122226] *** VALIDATE nfs *** [ 3.123722] *** VALIDATE nfs4 *** [ 3.125873] pstore: using deflate compression [ 3.129905] Platform Keyring initialized [ 3.237124] NET: Registered protocol family 38 [ 3.238776] Key type asymmetric registered [ 3.240407] Asymmetric key parser 'x509' registered [ 3.242342] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.245428] io scheduler mq-deadline registered [ 3.246882] io scheduler kyber registered [ 3.248488] io scheduler bfq registered [ 3.250179] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.252759] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.255193] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.258835] ACPI: Power Button [PWRF] [ 3.264877] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.272102] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.283031] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.293269] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.305489] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.333525] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.363278] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.367871] Non-volatile memory driver v1.3 [ 3.369543] Linux agpgart interface v0.103 [ 3.407605] virtio_blk virtio1: [vda] 134608 512-byte logical blocks (68.9 MB/65.7 MiB) [ 3.410963] vda: detected capacity change from 0 to 68919296 [ 3.430787] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.434567] vdb: detected capacity change from 0 to 1073741824 [ 3.450169] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.453716] vdc: detected capacity change from 0 to 2621440000 [ 3.471390] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.474763] vdd: detected capacity change from 0 to 2621440000 [ 3.489835] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.493028] vde: detected capacity change from 0 to 4294967296 [ 3.510989] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.515412] vdf: detected capacity change from 0 to 4294967296 [ 3.524740] libphy: Fixed MDIO Bus: probed [ 3.533869] usbcore: registered new interface driver usbserial_generic [ 3.537045] usbserial: USB Serial support registered for generic [ 3.539874] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.544385] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.545972] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.549196] mousedev: PS/2 mouse device common for all mice [ 3.553492] rtc_cmos 00:05: RTC can wake from S4 [ 3.553889] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.557177] rtc_cmos 00:05: registered as rtc0 [ 3.561416] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.563131] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.565292] intel_pstate: CPU model not supported [ 3.571572] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.577745] hid: raw HID events driver (C) Jiri Kosina [ 3.580054] usbcore: registered new interface driver usbhid [ 3.582126] usbhid: USB HID core driver [ 3.583675] drop_monitor: Initializing network drop monitor service [ 3.586769] Initializing XFRM netlink socket [ 3.589294] NET: Registered protocol family 10 [ 3.592472] Segment Routing with IPv6 [ 3.594235] NET: Registered protocol family 17 [ 3.596463] mpls_gso: MPLS GSO support [ 3.601690] RAS: Correctable Errors collector initialized. [ 3.603979] AVX version of gcm_enc/dec engaged. [ 3.605927] AES CTR mode by8 optimization enabled [ 3.688161] sched_clock: Marking stable (3688136546, 0)->(4633945500, -945808954) [ 3.692502] registered taskstats version 1 [ 3.695093] Loading compiled-in X.509 certificates [ 3.701336] zswap: loaded using pool lzo/zbud [ 3.727085] Key type big_key registered [ 3.739740] Key type encrypted registered [ 3.741323] ima: No TPM chip found, activating TPM-bypass! [ 3.743571] ima: Allocated hash algorithm: sha1 [ 3.745370] ima: No architecture policies found [ 3.748549] evm: Initialising EVM extended attributes: [ 3.750491] evm: security.selinux [ 3.751842] evm: security.ima [ 3.753041] evm: security.capability [ 3.754387] evm: HMAC attrs: 0x1 [ 3.757105] rtc_cmos 00:05: setting system clock to 2026-04-29 21:05:13 UTC (1777496713) [ 3.763804] debug: unmapping init [mem 0xffffffffa7603000-0xffffffffa77fffff] [ 3.767023] debug: unmapping init [mem 0xffffffffa6382000-0xffffffffa6658fff] [ 3.776070] Write protecting the kernel read-only data: 28672k [ 3.779791] debug: unmapping init [mem 0xffffffffa4a03000-0xffffffffa4bfffff] [ 3.782914] debug: unmapping init [mem 0xffffffffa5314000-0xffffffffa53fffff] [ 3.824347] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.832742] systemd[1]: Detected virtualization kvm. [ 3.834393] systemd[1]: Detected architecture x86-64. [ 3.836236] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.864820] systemd[1]: No hostname configured. [ 3.866054] systemd[1]: Set hostname to . [ 3.867761] random: systemd: uninitialized urandom read (16 bytes read) [ 3.870346] systemd[1]: Initializing machine ID from random generator. [ 3.921409] random: ln: uninitialized urandom read (6 bytes read) [ 4.015423] random: systemd: uninitialized urandom read (16 bytes read) [ 4.018278] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.023900] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.030983] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.729853] device-mapper: uevent: version 1.0.3 [ 4.731638] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.449573] random: fast init done [ 5.458207] virtio_net virtio0 ens2: renamed from eth0 [ 5.530359] scsi host0: ata_piix [ 5.538160] scsi host1: ata_piix [ 5.540582] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.542702] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.281675] random: crng init done [ 10.289253] random: 7 urandom warning(s) missed due to ratelimiting [ 10.463941] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.214661] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.316690] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.458854] SELinux: Disabled at runtime. [ 17.572499] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.598271] systemd[1]: Detected virtualization kvm. [ 17.600980] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.612760] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.621911] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.649317] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.665537] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.674879] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.704376] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.722334] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-f[ 20.351475] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS s-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.630776] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.625684] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.828822] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.589439] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.690473] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 28.897704] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 29.988457] NFS: Registering the id_resolver key type [ 29.992951] Key type id_resolver registered [ 29.996557] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg333-server login: [ 103.658708] libcfs: loading out-of-tree module taints kernel. [ 103.692258] Key type ._llcrypt registered [ 103.696417] Key type .llcrypt registered [ 103.865530] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 126.190565] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 128.260444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.281141] alg: No test for adler32 (adler32-zlib) [ 130.008343] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 131.342763] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 133.312168] Key type lgssc registered [ 135.458341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.977709] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 165.992597] hrtimer: interrupt took 2869803 ns [ 199.488827] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 215.788875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 215.879190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.266733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 217.327568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 217.487659] Lustre: lustre-MDT0000: new disk, initializing [ 217.697628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.745205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 223.592422] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.565622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.703858] Lustre: 6513:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 239.765571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 239.769749] Lustre: Skipped 1 previous similar message [ 239.862345] Lustre: lustre-MDT0001: new disk, initializing [ 239.945062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.020341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 240.038255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 244.408041] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.336037] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 260.747918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.149964] Lustre: lustre-OST0000: new disk, initializing [ 261.158982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 261.255897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 263.370458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 263.383000] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 263.516915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 268.480555] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 284.224773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 284.376236] Lustre: lustre-OST0001: new disk, initializing [ 284.381721] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 284.471959] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 291.226306] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.960044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 293.979723] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 294.082528] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 304.292594] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 313.716872] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 320.151283] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing check_logdir /tmp/testlogs/ [ 325.036409] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing yml_node [ 329.019703] Lustre: DEBUG MARKER: Client: 2.17.52.93 [ 331.601197] Lustre: DEBUG MARKER: MDS: 2.17.52.93 [ 334.411331] Lustre: DEBUG MARKER: OSS: 2.17.52.93 [ 336.490543] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Apr 29 17:10:43 EDT 2026 [ 357.556839] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 359.447880] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 376.289401] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.289702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.305333] Lustre: Skipped 1 previous similar message [ 376.316520] Lustre: Skipped 3 previous similar messages [ 381.538088] Lustre: server umount lustre-MDT0000 complete [ 385.584924] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497095 with bad export cookie 2170194736372036049 [ 385.596607] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 385.604857] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 386.531414] LustreError: 7722:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.531885] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.535758] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 386.561897] LustreError: 7722:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 386.611048] Lustre: Skipped 3 previous similar messages [ 390.560887] LustreError: 6520:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.562186] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 390.592965] Lustre: Skipped 2 previous similar messages [ 391.914296] Lustre: server umount lustre-MDT0001 complete [ 395.762499] Lustre: server umount lustre-OST0000 complete [ 400.216291] Lustre: server umount lustre-OST0001 complete [ 407.616483] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 417.304268] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 457.650603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.968333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 458.035841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 458.113393] Lustre: lustre-MDT0000: new disk, initializing [ 458.228370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 458.245306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 462.824077] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 473.321147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.474375] Lustre: 14992:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 473.550868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 473.559344] Lustre: Skipped 1 previous similar message [ 473.630881] Lustre: lustre-MDT0001: new disk, initializing [ 473.731206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 473.771116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 473.790194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 478.542284] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 486.694815] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 491.755839] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 499.567456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 499.915709] Lustre: lustre-OST0000: new disk, initializing [ 499.919828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 500.049322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.158297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 501.180503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 501.304163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 506.802237] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 515.537985] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 520.523168] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 520.719499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 525.237527] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 525.466693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 531.938165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 531.941876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.947728] LustreError: Skipped 1 previous similar message [ 531.950242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 531.958922] Lustre: Skipped 1 previous similar message [ 531.994562] Lustre: Skipped 1 previous similar message [ 537.066848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.077504] Lustre: Skipped 1 previous similar message [ 542.182963] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 542.393216] Lustre: server umount lustre-OST0000 complete [ 547.321272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.330048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 547.347932] Lustre: Skipped 1 previous similar message [ 547.368947] Lustre: Skipped 3 previous similar messages [ 552.829550] Lustre: server umount lustre-MDT0000 complete [ 556.691319] LustreError: 14985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497266 with bad export cookie 2170194736372037582 [ 556.696410] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.702474] LustreError: 14985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 557.094128] Lustre: server umount lustre-MDT0001 complete [ 563.425518] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 17:14:30 (1777497270) [ 582.664695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.039039] LustreError: 19468:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.055263] LustreError: 19468:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 583.107880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.807233] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 588.258408] LustreError: 19469:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.358144] LustreError: 19468:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.799989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.135289] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 604.414307] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 607.741704] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 609.835909] Lustre: Failing over lustre-MDT0000 [ 610.105587] Lustre: server umount lustre-MDT0000 complete [ 610.785602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 610.793102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.818133] LustreError: 19468:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.217641] Lustre: 3663:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777497322/real 1777497322] req@ffff8b6ec6056d80 x1863840527708288/t0(0) o400->MGC192.168.203.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777497338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 629.250696] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 629.256248] LustreError: 19469:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.262503] LustreError: 19469:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [ 629.446911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.432990] LustreError: 3661:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b6fc7118e00 x1863840527710720/t0(0) o250->MGC192.168.203.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 638.862618] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 639.800616] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 639.806357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 639.853588] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 644.802225] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 655.148304] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 659.201111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.228657] Lustre: Skipped 1 previous similar message [ 659.236372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 659.240974] Lustre: Skipped 3 previous similar messages [ 665.478937] Lustre: server umount lustre-MDT0000 complete [ 669.671127] LustreError: 19469:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 669.688971] LustreError: 19469:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 669.824944] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497379 with bad export cookie 2170194736372038996 [ 669.828693] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.847192] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 670.444694] Lustre: server umount lustre-MDT0001 complete [ 679.598092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.214632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.216894] Lustre: Skipped 2 previous similar messages [ 684.780545] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 693.621907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.754957] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 704.812398] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 709.206242] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 711.535970] Lustre: Failing over lustre-MDT0000 [ 711.896214] Lustre: server umount lustre-MDT0000 complete [ 714.722304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.724909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 714.731339] Lustre: Skipped 2 previous similar messages [ 731.108115] Lustre: 3662:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777497424/real 1777497424] req@ffff8b6ec6055f80 x1863840527738112/t0(0) o400->MGC192.168.203.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777497440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 731.140655] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 732.095795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.226135] LustreError: 23439:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 736.249425] LustreError: 23439:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 14 previous similar messages [ 740.792161] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 745.615723] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 745.965291] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 745.977261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 745.992895] Lustre: Skipped 1 previous similar message [ 746.033457] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 754.618277] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 761.314879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 761.315800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.327659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 761.345567] Lustre: Skipped 1 previous similar message [ 761.363243] Lustre: Skipped 3 previous similar messages [ 765.662627] Lustre: server umount lustre-MDT0000 complete [ 771.773041] LustreError: 22446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497481 with bad export cookie 2170194736372039934 [ 771.802284] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 771.808031] LustreError: 22446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 772.430688] Lustre: server umount lustre-MDT0001 complete [ 785.504681] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:18:12 (1777497492) [ 802.014269] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 811.712153] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 851.865633] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 863.237975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.561618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 863.600944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 863.705597] Lustre: lustre-MDT0000: new disk, initializing [ 863.906982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.921061] Lustre: Skipped 2 previous similar messages [ 863.950657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 868.849855] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 880.471678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.589046] Lustre: 28449:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 880.880370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 885.440406] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 893.632813] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 898.443910] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 906.463392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 908.327696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 908.363836] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 912.724691] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 913.907062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 913.927756] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 913.937194] Lustre: Skipped 1 previous similar message [ 913.941153] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 920.724362] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 926.152406] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 949.218389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 949.225572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.258193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 949.266599] Lustre: Skipped 2 previous similar messages [ 954.622401] Lustre: server umount lustre-OST0000 complete [ 964.764431] Lustre: server umount lustre-MDT0000 complete [ 965.105546] LustreError: 28460:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.118786] LustreError: 28460:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 969.317398] LustreError: 28440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497679 with bad export cookie 2170194736372040340 [ 969.332325] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 969.337613] LustreError: 28440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 969.865900] Lustre: server umount lustre-MDT0001 complete [ 981.978197] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 985.204275] Key type lgssc unregistered [ 985.538617] LNet: 32041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.545096] LNetError: 32041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.558580] LNet: Removed LNI 192.168.203.133@tcp [ 986.524261] Key type .llcrypt unregistered [ 986.526115] Key type ._llcrypt unregistered [ 1012.860996] Key type ._llcrypt registered [ 1012.864562] Key type .llcrypt registered [ 1012.998115] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1014.199876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1014.219704] alg: No test for adler32 (adler32-zlib) [ 1015.380956] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1015.611108] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1017.376463] Key type lgssc registered [ 1018.843193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1026.900137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1026.915861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.447883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.569467] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1038.822347] LustreError: 33445:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.844766] LustreError: 33445:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1043.169077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.594626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.604363] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1055.359584] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1059.387652] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1066.672847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1067.061558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.124090] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1074.279627] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1079.915547] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1095.618867] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:23:22 (1777497802) [ 1097.599463] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1099.288854] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1107.107610] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:23:33 (1777497813) [ 1116.568596] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:23:43 (1777497823) [ 1117.480898] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1128.347442] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1128.360340] Lustre: Skipped 2 previous similar messages [ 1130.688301] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:23:57 (1777497837) [ 1133.222680] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1133.229431] Lustre: Skipped 2 previous similar messages [ 1142.034435] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1142.041941] Lustre: Skipped 4 previous similar messages [ 1158.796775] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:24:25 (1777497865) [ 1162.122571] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1165.942846] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1175.677949] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:24:42 (1777497882) [ 1177.694892] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1177.703872] Lustre: Skipped 6 previous similar messages [ 1191.420801] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:24:58 (1777497898) [ 1207.267394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1207.270698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.273385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1207.285542] LustreError: Skipped 1 previous similar message [ 1207.325136] Lustre: Skipped 1 previous similar message [ 1211.843675] Lustre: server umount lustre-OST0000 complete [ 1217.506637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.525048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1217.530441] Lustre: Skipped 1 previous similar message [ 1222.632465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1222.642547] Lustre: Skipped 1 previous similar message [ 1227.751608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1227.761928] Lustre: Skipped 1 previous similar message [ 1230.304137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1230.546881] Lustre: server umount lustre-MDT0000 complete [ 1232.873351] LustreError: 33445:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.151795] LustreError: 33425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497943 with bad export cookie 11773826910297724508 [ 1234.163025] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1234.169087] LustreError: 33425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1234.599335] Lustre: server umount lustre-MDT0001 complete [ 1246.613282] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1249.537502] Key type lgssc unregistered [ 1249.822871] LNet: 40956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.834159] LNetError: 40956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.867908] LNet: Removed LNI 192.168.203.133@tcp [ 1250.886364] Key type .llcrypt unregistered [ 1250.891713] Key type ._llcrypt unregistered [ 1271.105165] Key type ._llcrypt registered [ 1271.107102] Key type .llcrypt registered [ 1271.208290] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 1287.598666] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1289.389492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1289.471114] alg: No test for adler32 (adler32-zlib) [ 1290.675638] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1291.013945] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1292.736543] Key type lgssc registered [ 1293.913485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1331.910636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1331.949466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.293948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1333.338891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1333.420354] Lustre: lustre-MDT0000: new disk, initializing [ 1333.546788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.569125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1338.097412] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1348.953852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.097284] Lustre: 44081:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1349.191796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1349.200344] Lustre: Skipped 1 previous similar message [ 1349.299154] Lustre: lustre-MDT0001: new disk, initializing [ 1349.397266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.440134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1349.456690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1353.882802] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1361.956287] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1365.867513] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1372.668896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1373.032969] Lustre: lustre-OST0000: new disk, initializing [ 1373.041227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1373.184298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.337764] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1383.474669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1383.487323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1383.544357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1387.050592] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1392.026597] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1392.296949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1396.716901] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1396.908366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1398.759349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1398.766671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.777716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1398.788842] Lustre: Skipped 1 previous similar message [ 1403.882652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1403.891797] Lustre: Skipped 2 previous similar messages [ 1404.738983] Lustre: server umount lustre-OST0000 complete [ 1408.997485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1409.004759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1409.017700] Lustre: Skipped 1 previous similar message [ 1412.069354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1412.079934] Lustre: Skipped 2 previous similar messages [ 1414.575274] Lustre: server umount lustre-MDT0000 complete [ 1418.341678] LustreError: 44858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498128 with bad export cookie 3596807346101948397 [ 1418.352506] LustreError: 44858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1418.356916] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1418.766769] Lustre: server umount lustre-MDT0001 complete [ 1428.753308] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:28:55 (1777498135) [ 1436.253967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.950917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1441.006772] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1442.275228] LustreError: 48136:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.302946] LustreError: 48136:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1446.380840] LustreError: 48137:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.992546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.423379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1453.565121] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1458.771546] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1461.930415] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1468.265842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1468.643138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.458673] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1480.009141] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1488.902407] LustreError: 50326:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1489.896076] Lustre: Modifying parameter general.timeout=1 in log params [ 1490.898325] Lustre: Modifying parameter general.timeout=2 in log params [ 1493.301975] Lustre: Modifying parameter general.timeout=5 in log params [ 1493.307361] Lustre: Skipped 2 previous similar messages [ 1498.055995] Lustre: Modifying parameter general.timeout=10 in log params [ 1498.060932] Lustre: Skipped 4 previous similar messages [ 1506.128366] Lustre: Modifying parameter general.timeout=19 in log params [ 1506.132123] Lustre: Skipped 8 previous similar messages [ 1522.173214] Lustre: Modifying parameter general.timeout=40 in log params [ 1522.178955] Lustre: Skipped 20 previous similar messages [ 1554.580820] Lustre: Modifying parameter general.timeout=84 in log params [ 1554.585485] Lustre: Skipped 43 previous similar messages [ 1596.071967] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:31:43 (1777498303) [ 1598.477340] Lustre: 57258:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1600.070207] Lustre: 57356:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1603.044821] Lustre: 57553:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1603.054664] Lustre: 57553:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1619.515341] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1619.519383] Lustre: Skipped 64 previous similar messages [ 1626.124830] Lustre: 59126:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1626.131985] Lustre: 59126:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1626.142803] LustreError: 59126:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1629.392798] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:32:16 (1777498336) [ 1638.884807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1638.894525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1638.914440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.925913] Lustre: Skipped 2 previous similar messages [ 1644.659479] Lustre: server umount lustre-MDT0000 complete [ 1647.331938] LustreError: 57714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498357 with bad export cookie 3596807346101949279 [ 1647.336487] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1647.343216] LustreError: 57714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1647.580256] Lustre: server umount lustre-MDT0001 complete [ 1661.105493] Lustre: server umount lustre-OST0000 complete [ 1681.799360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1682.012080] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1682.187380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.127893] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1691.138610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.240691] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1691.262859] Lustre: 61660:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1694.145351] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1698.538971] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1701.444506] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1706.892589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1707.089045] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1711.924073] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1716.183944] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1717.399158] Lustre: 63884:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1722.850683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1722.856713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.862161] LustreError: Skipped 1 previous similar message [ 1722.870179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1722.877828] Lustre: Skipped 3 previous similar messages [ 1722.890710] Lustre: Skipped 7 previous similar messages [ 1727.971325] LustreError: 63690:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1727.987503] LustreError: 63690:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1728.041368] Lustre: server umount lustre-OST0000 complete [ 1732.578279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1732.583647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.329945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1743.340334] Lustre: Skipped 6 previous similar messages [ 1745.376119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1745.494852] Lustre: server umount lustre-MDT0000 complete [ 1748.055908] LustreError: 61652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498457 with bad export cookie 3596807346101952142 [ 1748.059825] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1748.065400] LustreError: 61652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1748.294602] Lustre: server umount lustre-MDT0001 complete [ 1757.062240] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1759.040443] Key type lgssc unregistered [ 1759.281706] LNet: 65073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.286960] LNetError: 65073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.306826] LNet: Removed LNI 192.168.203.133@tcp [ 1759.868351] Key type .llcrypt unregistered [ 1759.873417] Key type ._llcrypt unregistered [ 1779.177657] Key type ._llcrypt registered [ 1779.179650] Key type .llcrypt registered [ 1779.243618] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1779.997671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1780.012377] alg: No test for adler32 (adler32-zlib) [ 1781.003870] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1781.176057] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1782.808250] Key type lgssc registered [ 1783.610612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1789.519483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1789.550722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.893735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.680474] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1799.521405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.018811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1803.005753] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1807.205905] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1809.828954] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1814.510499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1814.888800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.953917] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1822.499878] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1831.343631] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1840.610138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.612325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1840.623342] Lustre: Skipped 2 previous similar messages [ 1840.628333] Lustre: Skipped 1 previous similar message [ 1845.734286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1845.737953] Lustre: Skipped 1 previous similar message [ 1850.852165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.856839] Lustre: Skipped 2 previous similar messages [ 1851.361075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1851.502632] Lustre: server umount lustre-MDT0000 complete [ 1854.145540] LustreError: 66455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498563 with bad export cookie 7367285513387407187 [ 1854.152894] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1854.157537] LustreError: 66455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1854.362941] Lustre: server umount lustre-MDT0001 complete [ 1867.577462] Lustre: server umount lustre-OST0000 complete [ 1883.876091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.073935] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1884.103187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1884.256875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1886.457659] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1891.578363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.687287] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1891.722588] Lustre: 71005:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1892.037891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1894.866992] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1898.567840] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1900.972885] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1905.141278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1905.273538] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1905.522249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.377971] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1912.845851] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1913.957557] Lustre: 73240:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1915.071644] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1922.533130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1922.537714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.549363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1922.553790] Lustre: Skipped 1 previous similar message [ 1924.257886] Lustre: server umount lustre-OST0000 complete [ 1927.651846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.652633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.652759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1927.652763] LustreError: Skipped 1 previous similar message [ 1927.666867] Lustre: Skipped 2 previous similar messages [ 1932.985947] Lustre: server umount lustre-MDT0000 complete [ 1935.526880] LustreError: 72645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498645 with bad export cookie 7367285513387408020 [ 1935.530549] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1935.542029] LustreError: 72645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1935.784788] Lustre: server umount lustre-MDT0001 complete [ 1943.537456] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1945.511350] Key type lgssc unregistered [ 1945.725806] LNet: 74428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.732363] LNetError: 74428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.743792] LNet: Removed LNI 192.168.203.133@tcp [ 1946.370577] Key type .llcrypt unregistered [ 1946.372626] Key type ._llcrypt unregistered [ 1963.922978] Key type ._llcrypt registered [ 1963.925173] Key type .llcrypt registered [ 1963.997769] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1964.614721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.626623] alg: No test for adler32 (adler32-zlib) [ 1965.551843] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1965.713082] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1967.328325] Key type lgssc registered [ 1968.158702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1972.816930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1972.828110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.220476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.039959] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1983.021291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.378062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1985.828914] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1989.145911] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1991.527381] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1995.932207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1996.325513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.116844] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2003.677085] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2011.498324] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2012.067333] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2013.144761] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2013.149713] Lustre: Skipped 1 previous similar message [ 2015.618397] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2015.624064] Lustre: Skipped 3 previous similar messages [ 2020.229088] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2020.238863] Lustre: Skipped 6 previous similar messages [ 2028.607984] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2028.615299] Lustre: Skipped 13 previous similar messages [ 2044.870929] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2044.874421] Lustre: Skipped 26 previous similar messages [ 2075.253090] Lustre: 83545:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2076.894748] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:39:44 (1777498784) [ 2077.895850] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2079.003982] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:39:46 (1777498786) [ 2081.250054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2081.259613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.275930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2084.320962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2084.324519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.327486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2084.333395] Lustre: Skipped 1 previous similar message [ 2087.269463] Lustre: server umount lustre-OST0000 complete [ 2089.442192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.453429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.456868] Lustre: Skipped 1 previous similar message [ 2094.561458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.566793] Lustre: Skipped 1 previous similar message [ 2095.341293] Lustre: server umount lustre-MDT0000 complete [ 2097.433783] LustreError: 75786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498807 with bad export cookie 8559497365929567095 [ 2097.437233] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2097.440700] LustreError: 75786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2097.641656] Lustre: server umount lustre-MDT0001 complete [ 2103.538724] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2105.144493] Key type lgssc unregistered [ 2105.345501] LNet: 84778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.351587] LNetError: 84778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.406340] LNet: Removed LNI 192.168.203.133@tcp [ 2106.817277] Key type .llcrypt unregistered [ 2106.820311] Key type ._llcrypt unregistered [ 2108.768860] Key type ._llcrypt registered [ 2108.770285] Key type .llcrypt registered [ 2108.827270] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2111.029375] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules [ 2111.469383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2111.501176] alg: No test for adler32 (adler32-zlib) [ 2112.436822] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2112.599385] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2114.232166] Key type lgssc registered [ 2114.961812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2115.714680] LustreError: 85897:0:(lu_object.c:1479:lu_context_key_register()) cfs_race id 60d sleeping [ 2126.266842] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2132.285895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2138.307972] LustreError: 85897:0:(lu_object.c:1479:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2138.500201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.209258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.506323] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2150.414282] LustreError: 86904:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.449902] LustreError: 86904:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2155.497132] LustreError: 86903:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.613408] LustreError: 86904:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.182504] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:41:07 (1777498867) [ 2165.731628] LustreError: 86903:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.947527] LustreError: 86935:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2168.957666] LustreError: 86935:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2168.966044] LustreError: 86935:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 28, retries 0, failed: rc = -5 [ 2169.649323] Lustre: server umount lustre-MDT0000 complete [ 2187.954889] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2191.785788] Key type lgssc unregistered [ 2192.319852] LNet: 88395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2192.342521] LNetError: 88395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2192.368637] LNet: Removed LNI 192.168.203.133@tcp [ 2193.699759] Key type .llcrypt unregistered [ 2193.708175] Key type ._llcrypt unregistered [ 2213.577382] Key type ._llcrypt registered [ 2213.578967] Key type .llcrypt registered [ 2213.685347] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2215.125654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2215.158161] alg: No test for adler32 (adler32-zlib) [ 2216.275929] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2216.545337] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2218.248346] Key type lgssc registered [ 2219.622964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2227.496344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2227.510948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.053512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2232.680497] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2238.434782] LustreError: 89598:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.451400] LustreError: 89598:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2239.998139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.628449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2244.514773] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2249.707948] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2253.186673] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2259.460588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.100970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.744632] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2269.697996] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2315.075036] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:43:42 (1777499022) [ 2316.353567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2317.884983] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:43:45 (1777499025) [ 2322.404338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2322.408375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.418954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.918044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.922378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.931652] Lustre: Skipped 1 previous similar message [ 2322.945454] Lustre: Skipped 1 previous similar message [ 2327.544524] Lustre: server umount lustre-MDT0000 complete [ 2328.035919] LustreError: 89594:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.051346] LustreError: 89594:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2330.350973] LustreError: 89577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499040 with bad export cookie 1201029407151389419 [ 2330.358913] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2330.364395] LustreError: 89577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2330.540839] Lustre: server umount lustre-MDT0001 complete [ 2343.840327] Lustre: server umount lustre-OST0000 complete [ 2352.405721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2352.715427] LustreError: 93529:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.836893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2355.683444] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2358.242436] LustreError: 93530:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.151736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2361.732576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2365.021683] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2369.934588] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2373.470234] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2385.077921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2385.301222] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2385.316322] LustreError: 93518:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2385.364397] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2385.385553] LustreError: 95386:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2385.396985] LustreError: 95386:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2385.413539] LustreError: 95386:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2385.505520] Lustre: server umount lustre-OST0000 complete [ 2385.516097] LustreError: 95386:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2392.915450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2393.123206] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2393.136271] LustreError: 93517:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2393.145671] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2393.152786] LustreError: 95848:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2393.165177] LustreError: 95848:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2393.173036] LustreError: 95848:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2393.247247] Lustre: server umount lustre-OST0000 complete [ 2393.252311] LustreError: 95848:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2401.303047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2401.484720] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2401.493822] Lustre: Found index 0 for lustre-OST0000, updating log [ 2401.498936] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2401.575465] Lustre: lustre-OST0000: new disk, initializing [ 2401.579198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2401.811829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.496983] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2411.000117] Lustre: Failing over lustre-OST0000 [ 2411.197705] Lustre: server umount lustre-OST0000 complete [ 2417.124415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2417.134644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2417.137217] Lustre: Skipped 1 previous similar message [ 2417.146331] Lustre: Skipped 1 previous similar message [ 2420.803906] Lustre: server umount lustre-MDT0000 complete [ 2422.242811] LustreError: 95820:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.265986] LustreError: 95820:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2424.199689] LustreError: 94252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499133 with bad export cookie 1201029407151392163 [ 2424.202535] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.209246] LustreError: 94252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2433.613808] Lustre: server umount lustre-MDT0001 complete [ 2440.981971] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:45:48 (1777499148) [ 2454.126915] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2461.289056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.533339] LustreError: 98707:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.604298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.661410] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2469.964813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2473.012635] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2475.155590] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2480.443125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2481.072525] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2485.327277] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2487.212907] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2487.219223] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2487.228930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2491.288981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2491.345932] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2491.425170] Lustre: lustre-OST0001: new disk, initializing [ 2491.431327] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2491.585940] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2491.596812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2492.721250] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2492.764247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2492.769516] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2492.812667] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2494.827781] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 2494.846146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2494.851132] LustreError: Skipped 1 previous similar message [ 2494.856780] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2494.857038] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2495.584344] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2501.154905] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2504.279146] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2507.238070] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2507.241197] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.251150] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2513.023627] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.33@tcp (stopping) [ 2513.028165] Lustre: Skipped 4 previous similar messages [ 2513.302171] Lustre: server umount lustre-MDT0001 complete [ 2515.428093] LustreError: 98707:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.438134] LustreError: 98707:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2519.756255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.871095] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2519.875359] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2519.921266] Lustre: 98697:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2520.476752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2520.485026] Lustre: Skipped 3 previous similar messages [ 2521.839749] LustreError: 102560:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2521.855109] LustreError: 102560:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2521.867277] LustreError: 98699:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2521.940510] LustreError: 102578:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.133@tcp: cfg command failed: rc = -17 [ 2521.945813] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2521.945813] [ 2523.492384] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2525.680221] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2530.748355] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 17:47:18 (1777499238) [ 2544.303443] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2558.208625] Lustre: 104536:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2574.631423] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2577.347600] Lustre: 105669:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2653.152955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2653.160832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2653.168571] Lustre: Skipped 2 previous similar messages [ 2653.177638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.931815] Lustre: server umount lustre-MDT0000 complete [ 2658.789953] LustreError: 98704:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.810121] LustreError: 98704:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 2660.596573] LustreError: 98687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499370 with bad export cookie 1201029407151393094 [ 2660.597156] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2660.606330] LustreError: 98687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2660.806115] Lustre: server umount lustre-MDT0001 complete [ 2674.228570] Lustre: server umount lustre-OST0000 complete [ 2686.625872] Lustre: server umount lustre-OST0001 complete [ 2691.377684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2699.592198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2706.106964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2713.440191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2727.376086] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2734.840109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.857974] Lustre: lustre-MDT0000: reset Object Index mappings [ 2735.227527] LustreError: 109604:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.318749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.442965] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2743.951565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.973101] Lustre: lustre-MDT0001: reset Object Index mappings [ 2747.016096] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2748.951900] Lustre: 110716:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2753.249964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.691813] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2759.168814] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 2761.207470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 2763.885234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2768.423848] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2769.400052] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2769.402381] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2774.428726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2776.830190] Lustre: 112556:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2816.213345] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:52:03 (1777499523) [ 2846.181297] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.189959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2846.191995] Lustre: Skipped 5 previous similar messages [ 2846.202100] Lustre: Skipped 6 previous similar messages [ 2850.022270] Lustre: server umount lustre-MDT0000 complete [ 2852.637064] LustreError: 109586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499562 with bad export cookie 1201029407151439966 [ 2852.640859] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2852.645850] LustreError: 109586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2859.052913] Lustre: server umount lustre-MDT0001 complete [ 2876.384250] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2876.441031] Lustre: server umount lustre-OST0000 complete [ 2893.280128] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2893.448493] Lustre: server umount lustre-OST0001 complete [ 2897.831126] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 2904.266215] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2926.458878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.636310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2926.659399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2926.712330] Lustre: lustre-MDT0000: new disk, initializing [ 2926.760889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.765293] Lustre: Skipped 3 previous similar messages [ 2926.777347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2929.042862] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2933.335641] Lustre: server umount lustre-MDT0000 complete [ 2938.942787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.138756] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2939.152623] Lustre: 116772:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2939.162491] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2941.802738] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2945.082148] Lustre: server umount lustre-MDT0000 complete [ 2951.181705] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:54:18 (1777499658) [ 2952.213355] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2953.209837] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:54:20 (1777499660) [ 2963.972234] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2971.444826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.912050] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2981.299582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2981.371121] Lustre: 118786:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2981.414729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2981.419629] Lustre: Skipped 1 previous similar message [ 2981.498662] Lustre: lustre-MDT0001: new disk, initializing [ 2981.604768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2981.610290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2984.508796] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2987.957353] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2992.622351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2992.800774] Lustre: lustre-OST0000: new disk, initializing [ 2992.805274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2993.955068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2993.961220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2994.055706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2996.929187] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3004.542337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3004.714715] Lustre: lustre-OST0001: new disk, initializing [ 3004.718723] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3006.508790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3006.517115] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3006.566848] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3008.912644] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3015.894170] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3018.192559] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3314.520164] Lustre: 127567:0:(osd_internal.h:1447:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3314.525029] Lustre: 127567:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3314.529746] Lustre: 127567:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3314.534606] Lustre: 127567:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3314.538793] Lustre: 127567:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3314.541608] Lustre: 127567:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3462.559776] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:02:49 (1777500169) [ 3477.474391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3477.478975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3477.481551] Lustre: Skipped 5 previous similar messages [ 3477.484163] Lustre: Skipped 4 previous similar messages [ 3489.248230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3489.402765] Lustre: server umount lustre-MDT0000 complete [ 3491.322250] LustreError: 118778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500201 with bad export cookie 1201029407151505255 [ 3491.324099] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3491.330967] LustreError: 118778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3492.833126] LustreError: 119660:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3492.834266] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.845571] LustreError: 119660:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 14 previous similar messages [ 3492.860443] Lustre: Skipped 2 previous similar messages [ 3497.616518] Lustre: server umount lustre-MDT0001 complete [ 3514.336108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3514.413164] Lustre: server umount lustre-OST0000 complete [ 3522.829737] Lustre: server umount lustre-OST0001 complete [ 3526.406496] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 3531.751771] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3551.132622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.274779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3551.295946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3551.374243] Lustre: lustre-MDT0000: new disk, initializing [ 3551.430370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.433638] Lustre: Skipped 5 previous similar messages [ 3551.446961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3553.701832] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3559.470254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.527042] Lustre: 130574:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3559.729088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3561.919740] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3566.584718] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3568.694892] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3572.668318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3574.738981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3574.748161] Lustre: Skipped 1 previous similar message [ 3574.757181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3574.796905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3576.094611] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3580.378757] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3585.610231] Lustre: lustre-MDD0000: changelog on [ 3587.124870] Lustre: lustre-MDD0001: changelog on [ 4276.604182] Lustre: lustre-MDD0001: changelog off [ 4277.405957] Lustre: 132875:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4277.771688] Lustre: lustre-MDD0000: changelog off [ 4281.317456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4281.317659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.319615] LustreError: Skipped 1 previous similar message [ 4281.320144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4281.324313] Lustre: Skipped 1 previous similar message [ 4281.325632] Lustre: Skipped 12 previous similar messages [ 4285.782877] Lustre: server umount lustre-OST0000 complete [ 4291.553866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.554857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4291.560211] Lustre: Skipped 1 previous similar message [ 4291.563382] Lustre: Skipped 1 previous similar message [ 4293.378448] Lustre: server umount lustre-MDT0000 complete [ 4294.903451] LustreError: 130566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501004 with bad export cookie 1201029407152278580 [ 4294.905746] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.907700] LustreError: 130566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4295.041125] Lustre: server umount lustre-MDT0001 complete [ 4299.351984] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4300.415986] Key type lgssc unregistered [ 4300.541340] LNet: 141001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4300.544059] LNetError: 141001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4300.550269] LNet: Removed LNI 192.168.203.133@tcp [ 4300.857093] Key type .llcrypt unregistered [ 4300.858647] Key type ._llcrypt unregistered [ 4307.719370] Key type ._llcrypt registered [ 4307.720619] Key type .llcrypt registered [ 4307.765412] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4314.238669] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4314.574261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4314.611038] alg: No test for adler32 (adler32-zlib) [ 4315.463607] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 4315.546856] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4317.128648] Key type lgssc registered [ 4317.538308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4329.935357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4329.942146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4331.028962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4331.040317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4331.077582] Lustre: lustre-MDT0000: new disk, initializing [ 4331.104153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4331.112487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4332.384349] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4336.709853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4336.753416] Lustre: 144140:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4336.779947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4336.782543] Lustre: Skipped 1 previous similar message [ 4336.825082] Lustre: lustre-MDT0001: new disk, initializing [ 4336.870000] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4336.886316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4336.891565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4338.376478] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4341.728950] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4343.085480] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4345.483082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4345.588399] Lustre: lustre-OST0000: new disk, initializing [ 4345.590065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4345.620390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4347.580752] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4348.540580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4348.544205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4348.579313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4350.794316] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4353.100269] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_module obdecho/obdecho [ 4353.726865] LustreError: 146753:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4353.730529] LustreError: 146753:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4354.353434] LustreError: 146849:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 4359.138619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4359.139959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4359.142698] Lustre: Skipped 2 previous similar messages [ 4359.144205] Lustre: Skipped 1 previous similar message [ 4363.801842] Lustre: server umount lustre-MDT0000 complete [ 4365.155263] LustreError: 145880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501074 with bad export cookie 12171708497905239602 [ 4365.157919] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4365.159274] LustreError: 145880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4365.270118] Lustre: server umount lustre-MDT0001 complete [ 4376.923675] Lustre: server umount lustre-OST0000 complete [ 4383.986979] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4387.558753] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4401.152653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4401.244225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4401.256705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4401.295627] Lustre: lustre-MDT0000: new disk, initializing [ 4401.323186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.331160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4402.796413] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4407.513640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.563773] Lustre: 150522:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4407.592645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4407.595117] Lustre: Skipped 1 previous similar message [ 4407.635424] Lustre: lustre-MDT0001: new disk, initializing [ 4407.674567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4407.690502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4407.696477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4409.300801] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4413.101887] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4414.752166] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4417.573963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4417.681351] Lustre: lustre-OST0000: new disk, initializing [ 4417.683461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4417.719702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4419.194658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4419.200767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4419.238602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4419.876913] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4423.162893] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4424.657797] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4424.746818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4426.261760] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4426.351280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4429.143181] Lustre: server umount lustre-OST0000 complete [ 4433.377806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4433.381511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.388446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4433.390791] Lustre: Skipped 1 previous similar message [ 4434.913599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.913851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4434.920930] Lustre: Skipped 1 previous similar message [ 4440.033148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4440.035969] Lustre: Skipped 1 previous similar message [ 4441.087980] Lustre: server umount lustre-MDT0000 complete [ 4442.628572] LustreError: 150514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501152 with bad export cookie 12171708497905240547 [ 4442.630679] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4442.634565] LustreError: 150514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4452.887880] Lustre: server umount lustre-MDT0001 complete [ 4454.261660] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:19:22 (1777501162) [ 4454.860490] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4455.569446] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:19:23 (1777501163) [ 4458.486089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.680912] LustreError: 154522:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4458.688603] LustreError: 154522:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 4458.713808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4460.373812] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4463.900428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4465.583306] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4467.624113] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4468.984503] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4471.352316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4473.551334] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4475.436234] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4489.186777] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.187652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4489.190718] Lustre: Skipped 1 previous similar message [ 4494.058369] Lustre: server umount lustre-MDT0000 complete [ 4494.306930] LustreError: 154523:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4494.312951] LustreError: 154523:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4495.435343] LustreError: 154502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501205 with bad export cookie 12171708497905241415 [ 4495.438384] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4495.440748] LustreError: 154502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4495.991792] LustreError: 154519:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.425314] LustreError: 154519:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.425499] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4499.437600] Lustre: Skipped 1 previous similar message [ 4499.440120] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4499.442886] Lustre: Skipped 4 previous similar messages [ 4501.619812] Lustre: server umount lustre-MDT0001 complete [ 4504.174894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4509.185401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.278350] LustreError: 158041:0:(llog.c:1646:llog_backup()) MGC192.168.203.133@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4509.281696] Lustre: 158041:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.203.133@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4525.686349] LustreError: 158046:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.696971] LustreError: 158046:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4535.925510] LustreError: 158047:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4535.932047] LustreError: 158047:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 4552.310189] LustreError: 158047:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4552.319210] LustreError: 158047:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 14 previous similar messages [ 4570.592342] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4570.595412] Lustre: 158041:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4585.952939] LustreError: 158066:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4585.957436] LustreError: 158066:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 19 previous similar messages [ 4611.559825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4611.563597] Lustre: Skipped 2 previous similar messages [ 4613.009951] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4615.651814] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4616.035147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4643.116627] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4645.461309] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4646.254224] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4646.888671] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4647.657884] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4648.260979] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4653.772295] Lustre: server umount lustre-MDT0000 complete [ 4655.006734] LustreError: 154504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501364 with bad export cookie 12171708497905247862 [ 4655.010908] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4655.012487] LustreError: 154504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4655.012582] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4655.017452] LustreError: 158114:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.026747] LustreError: 158114:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 40 previous similar messages [ 4660.193594] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4660.195271] Lustre: Skipped 2 previous similar messages [ 4661.163590] Lustre: server umount lustre-MDT0001 complete [ 4662.533663] Lustre: server umount lustre-OST0000 complete [ 4669.646274] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4672.904277] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4684.863067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4684.950576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4684.964301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4685.000747] Lustre: lustre-MDT0000: new disk, initializing [ 4685.024398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4685.026494] Lustre: Skipped 1 previous similar message [ 4685.032826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4686.269167] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4690.686540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4690.733799] Lustre: 162862:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4690.841971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4692.187866] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4695.620313] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4696.983342] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4699.363540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4700.536114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4700.568348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4701.489291] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4704.779592] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4706.172401] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4706.266042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4707.671721] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4707.751539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4710.882762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4710.884362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4710.885825] LustreError: Skipped 1 previous similar message [ 4710.886698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4710.890067] Lustre: Skipped 2 previous similar messages [ 4710.899012] Lustre: Skipped 1 previous similar message [ 4714.513751] Lustre: server umount lustre-OST0000 complete [ 4730.336491] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4730.401843] Lustre: server umount lustre-MDT0000 complete [ 4731.756111] LustreError: 162855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501441 with bad export cookie 12171708497905264690 [ 4731.758334] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4731.760558] LustreError: 162855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4731.876846] Lustre: server umount lustre-MDT0001 complete [ 4735.042351] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:24:03 (1777501443) [ 4737.326027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.869876] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4741.719211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4743.166699] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4744.955214] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4746.143472] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4748.548308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4750.712137] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4752.579462] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4772.709353] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:24:40 (1777501480) [ 4774.369093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4774.371844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4774.376769] Lustre: Skipped 2 previous similar messages [ 4777.440804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4777.444634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4777.446817] Lustre: Skipped 9 previous similar messages [ 4779.985037] Lustre: server umount lustre-OST0000 complete [ 4783.072939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4787.352293] Lustre: server umount lustre-MDT0000 complete [ 4788.704912] LustreError: 166885:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4788.710979] LustreError: 166885:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4788.728163] LustreError: 166846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501498 with bad export cookie 12171708497905265558 [ 4788.731558] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4788.733236] LustreError: 166846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4788.852234] Lustre: server umount lustre-MDT0001 complete [ 4792.732686] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4793.788402] Key type lgssc unregistered [ 4793.909428] LNet: 170258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4793.912886] LNetError: 170258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4793.922468] LNet: Removed LNI 192.168.203.133@tcp [ 4794.274571] Key type .llcrypt unregistered [ 4794.276640] Key type ._llcrypt unregistered [ 4802.359211] Key type ._llcrypt registered [ 4802.360555] Key type .llcrypt registered [ 4802.405738] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4802.916064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4802.928636] alg: No test for adler32 (adler32-zlib) [ 4803.773675] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 4803.847872] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4805.424139] Key type lgssc registered [ 4805.776078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4808.391414] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4808.398349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4823.840286] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4828.960166] LustreError: 171461:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 4834.080383] LustreError: 171461:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4834.085641] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4834.087543] LustreError: MGC192.168.203.133@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4834.091272] LustreError: 171461:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4834.094202] LustreError: 171461:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 4834.096885] LustreError: 171461:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4834.119804] Lustre: server umount lustre-OST0000 complete [ 4834.121428] LustreError: 171461:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4837.003758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4838.150276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.439995] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4842.345140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.473514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4843.757740] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4845.539973] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4846.717910] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4849.069119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4849.144867] LustreError: 173545:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4849.149075] Lustre: 173545:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.203.133@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4849.196449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4851.093446] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4852.934507] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4857.824790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4857.827328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.831852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4858.338467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4858.339035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4858.344233] Lustre: Skipped 2 previous similar messages [ 4859.799283] Lustre: server umount lustre-MDT0000 complete [ 4861.024271] LustreError: 171941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501570 with bad export cookie 6152031139752327011 [ 4861.027099] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4861.027551] LustreError: 171941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4861.141398] Lustre: server umount lustre-MDT0001 complete [ 4872.725618] Lustre: server umount lustre-OST0000 complete [ 4875.889683] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4879.155135] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4890.911418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.997073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4891.008142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4891.039740] Lustre: lustre-MDT0000: new disk, initializing [ 4891.061891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.068491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4892.341938] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4896.479038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4896.513909] Lustre: 176916:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4896.532566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4896.534322] Lustre: Skipped 1 previous similar message [ 4896.567320] Lustre: lustre-MDT0001: new disk, initializing [ 4896.595856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4896.606490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4896.609517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4897.866958] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4900.883675] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4902.056382] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4904.234203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4904.326483] Lustre: lustre-OST0000: new disk, initializing [ 4904.328351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4906.198082] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4908.218314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4908.221799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4908.245405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4909.284873] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4910.634967] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4910.705061] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4911.979655] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4912.058294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4913.636328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4913.636455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4913.637449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4913.640944] Lustre: Skipped 2 previous similar messages [ 4913.647437] LustreError: Skipped 1 previous similar message [ 4914.765560] Lustre: server umount lustre-OST0000 complete [ 4917.216780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4917.220092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4917.226761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4917.229240] Lustre: Skipped 1 previous similar message [ 4922.137588] Lustre: server umount lustre-MDT0000 complete [ 4923.341992] LustreError: 176909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501633 with bad export cookie 6152031139752327711 [ 4923.344446] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4923.345429] LustreError: 176909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4923.452728] Lustre: server umount lustre-MDT0001 complete [ 4926.131363] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:27:14 (1777501634) [ 4931.710427] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4932.823322] Key type lgssc unregistered [ 4932.951551] LNet: 181433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4932.954840] LNetError: 181433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4932.961428] LNet: Removed LNI 192.168.203.133@tcp [ 4933.295749] Key type .llcrypt unregistered [ 4933.297367] Key type ._llcrypt unregistered [ 4940.328108] Key type ._llcrypt registered [ 4940.329279] Key type .llcrypt registered [ 4940.375114] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4940.797050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4940.803429] alg: No test for adler32 (adler32-zlib) [ 4941.653147] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 4941.739606] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4943.320098] Key type lgssc registered [ 4943.665730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4946.142445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4946.149783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4947.295890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4948.609566] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4951.383276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.502383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4952.777132] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4954.450658] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4955.571823] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4958.027980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4958.094708] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4958.096442] LustreError: 184247:0:(llog.c:1687:llog_backup()) MGC192.168.203.133@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4958.100155] Lustre: 184247:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.203.133@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4958.136323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4959.892898] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4961.609694] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4966.880814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4966.883542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4966.888298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4967.393990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4967.394068] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4968.413256] Lustre: server umount lustre-MDT0000 complete [ 4969.609977] LustreError: 184274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501679 with bad export cookie 5609558046572119302 [ 4969.612986] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4969.613695] LustreError: 184274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4969.727093] Lustre: server umount lustre-MDT0001 complete [ 4981.199342] Lustre: server umount lustre-OST0000 complete [ 4983.898644] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4986.712063] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4997.739201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4997.815986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4997.825855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4997.856851] Lustre: lustre-MDT0000: new disk, initializing [ 4997.877618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4997.883335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4999.084698] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5002.995327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.032734] Lustre: 187618:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5003.049316] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5003.050476] Lustre: Skipped 1 previous similar message [ 5003.052287] LustreError: 188381:0:(llog.c:1687:llog_backup()) MGC192.168.203.133@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5003.055951] LustreError: 188381:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 5003.058620] Lustre: 188381:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.203.133@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5003.067353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5003.070844] Lustre: Skipped 1 previous similar message [ 5003.111156] Lustre: lustre-MDT0001: new disk, initializing [ 5003.143481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5003.154235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5003.157970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5004.442552] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5007.501345] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5008.692880] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5010.916279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5011.004826] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5011.006663] Lustre: Skipped 1 previous similar message [ 5011.008489] LustreError: 189337:0:(llog.c:1687:llog_backup()) MGC192.168.203.133@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5011.011356] LustreError: 189337:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 5011.014026] Lustre: 189337:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.203.133@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5011.017744] Lustre: 189337:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5011.026959] Lustre: lustre-OST0000: new disk, initializing [ 5011.028772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5012.861400] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5015.829101] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5017.168368] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5019.635996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5019.640035] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5019.669614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5020.297635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5021.637383] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5021.709881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5024.739166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.739249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5024.740418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5024.740424] Lustre: Skipped 2 previous similar messages [ 5024.742671] Lustre: Skipped 2 previous similar messages [ 5028.427032] Lustre: server umount lustre-OST0000 complete [ 5029.857149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5029.857609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5029.862053] Lustre: Skipped 1 previous similar message [ 5029.865905] Lustre: Skipped 2 previous similar messages [ 5034.977057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5034.980290] Lustre: Skipped 1 previous similar message [ 5035.670627] Lustre: server umount lustre-MDT0000 complete [ 5036.888906] LustreError: 188395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501746 with bad export cookie 5609558046572119995 [ 5036.891597] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5036.893131] LustreError: 188395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5036.997155] Lustre: server umount lustre-MDT0001 complete [ 5039.732048] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:29:07 (1777501747) [ 5044.979481] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5048.166637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.287600] LustreError: 192333:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5048.292653] LustreError: 192333:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5048.300142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5048.301774] Lustre: Skipped 1 previous similar message [ 5049.474570] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5052.157531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5053.537189] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5054.455450] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5056.609339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5058.399318] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5060.987673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5061.041388] Lustre: lustre-OST0001: new disk, initializing [ 5061.043365] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5062.794377] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5063.741349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5063.744954] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5063.767466] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5068.068023] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5074.654931] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5075.251715] Lustre: DEBUG MARKER: ADD OST3 [ 5076.849962] loop: module loaded [ 5076.865491] loop0: detected capacity change from 0 to 204800000 [ 5076.945355] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5076.948516] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5076.951845] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5078.826537] loop0: detected capacity change from 0 to 204800000 [ 5080.526357] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5080.580576] Lustre: lustre-OST0003: new disk, initializing [ 5080.581980] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5080.614128] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5080.616149] Lustre: Skipped 3 previous similar messages [ 5081.908170] Lustre: DEBUG MARKER: STOP OST3 [ 5082.465787] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5082.542978] Lustre: Failing over lustre-OST0003 [ 5082.575393] LustreError: 193784:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.577561] Lustre: server umount lustre-OST0003 complete [ 5086.762083] Lustre: DEBUG MARKER: START OST3 again [ 5087.358263] LustreError: 193784:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5087.364312] LustreError: 193784:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5088.871603] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5088.952282] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5090.363015] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5090.369668] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5090.723576] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5094.377573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5094.381079] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5094.390902] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5103.860594] Lustre: Failing over lustre-OST0003 [ 5103.901841] Lustre: server umount lustre-OST0003 complete [ 5104.609343] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5104.611391] LustreError: Skipped 1 previous similar message [ 5104.612744] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.615748] Lustre: Skipped 1 previous similar message [ 5104.615886] LustreError: 193783:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5104.621253] LustreError: 193783:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5109.731369] LustreError: 196801:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5109.737029] LustreError: 196801:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5118.944832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5118.947676] LustreError: Skipped 1 previous similar message [ 5118.949644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5118.957244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5119.972759] LustreError: 193784:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5119.977444] LustreError: 193784:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5122.464563] Lustre: server umount lustre-MDT0000 complete [ 5123.682253] LustreError: 192315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501833 with bad export cookie 5609558046572120870 [ 5123.685726] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5123.686326] LustreError: 192315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5129.897594] Lustre: server umount lustre-MDT0001 complete [ 5131.173024] Lustre: server umount lustre-OST0000 complete [ 5132.414278] Lustre: server umount lustre-OST0001 complete [ 5134.535652] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5137.160840] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5149.487911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5149.575781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5149.586447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5149.620927] Lustre: lustre-MDT0000: new disk, initializing [ 5149.643324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5149.646014] Lustre: Skipped 1 previous similar message [ 5149.652351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5150.945492] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5155.071791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5155.110578] Lustre: 201643:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5155.214599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5156.510195] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5159.619319] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5160.737831] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5162.877920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5164.280399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5164.313408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5164.823908] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5167.745853] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5168.959396] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5169.021544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5170.251448] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5170.322171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5174.753226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5174.757415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5174.762879] Lustre: Skipped 5 previous similar messages [ 5174.766036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5174.769195] Lustre: Skipped 9 previous similar messages [ 5177.034592] Lustre: server umount lustre-OST0000 complete [ 5184.408584] Lustre: server umount lustre-MDT0000 complete [ 5184.992887] LustreError: 201655:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5184.998699] LustreError: 201655:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 5185.572954] LustreError: 201634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501895 with bad export cookie 5609558046572123110 [ 5185.575858] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5185.577846] LustreError: 201634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5185.691827] Lustre: server umount lustre-MDT0001 complete [ 5186.797727] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:31:34 (1777501894) [ 5191.630752] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5194.611930] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5205.622778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5205.716914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5205.728059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5205.729568] Lustre: Skipped 3 previous similar messages [ 5205.761800] Lustre: lustre-MDT0000: new disk, initializing [ 5205.763138] Lustre: Skipped 2 previous similar messages [ 5205.790648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5205.793253] Lustre: Skipped 2 previous similar messages [ 5207.191260] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5211.221117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5211.255688] Lustre: 207804:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5211.350100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5212.574391] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5215.480236] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5216.624752] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5218.857297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5218.993325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.995663] Lustre: Skipped 4 previous similar messages [ 5220.565581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5220.708441] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5223.679594] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5224.984228] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5225.063609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5226.298174] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5226.368791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5231.074471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5231.075458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.077017] LustreError: Skipped 2 previous similar messages [ 5231.081106] Lustre: Skipped 3 previous similar messages [ 5231.083074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5231.086069] Lustre: Skipped 3 previous similar messages [ 5241.312124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5241.373617] Lustre: server umount lustre-OST0000 complete [ 5248.659435] Lustre: server umount lustre-MDT0000 complete [ 5249.835158] LustreError: 207797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501959 with bad export cookie 5609558046572123992 [ 5249.838614] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5249.839145] LustreError: 207797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5249.936372] Lustre: server umount lustre-MDT0001 complete [ 5252.605567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.758537] LustreError: 211614:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5253.972915] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5256.717914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.034790] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5259.675920] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5260.770291] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5262.824087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5264.781920] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5266.475957] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5274.592720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5279.053707] Lustre: server umount lustre-OST0000 complete [ 5286.355805] Lustre: server umount lustre-MDT0000 complete [ 5287.612130] LustreError: 211641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501997 with bad export cookie 5609558046572124874 [ 5287.614803] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5287.617079] LustreError: 211641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5287.736320] Lustre: server umount lustre-MDT0001 complete [ 5399.217840] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5400.194549] Key type lgssc unregistered [ 5400.321331] LNet: 214921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5400.323556] LNetError: 214921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5400.333265] LNet: Removed LNI 192.168.203.133@tcp [ 5400.628102] Key type .llcrypt unregistered [ 5400.629163] Key type ._llcrypt unregistered [ 5408.048558] Key type ._llcrypt registered [ 5408.049429] Key type .llcrypt registered [ 5408.079893] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5414.163622] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5414.466670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5414.474410] alg: No test for adler32 (adler32-zlib) [ 5415.319904] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 5415.400915] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5416.976089] Key type lgssc registered [ 5417.297194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5428.335513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5428.342461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5429.445835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5429.455189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5429.488736] Lustre: lustre-MDT0000: new disk, initializing [ 5429.515642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5429.523307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5430.944881] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5434.841563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.880237] Lustre: 218208:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5434.899553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5434.901561] Lustre: Skipped 1 previous similar message [ 5434.935964] Lustre: lustre-MDT0001: new disk, initializing [ 5434.966138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5434.978544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5434.981682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5436.186802] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5439.051674] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5440.249118] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5442.402285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5442.491123] Lustre: lustre-OST0000: new disk, initializing [ 5442.492807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5442.518984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5444.220504] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5446.005470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5446.008885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5446.039687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5447.140104] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5448.403126] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5448.475469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5449.721586] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5449.788020] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5451.233364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5451.234390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.236687] LustreError: Skipped 1 previous similar message [ 5451.240528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.243946] Lustre: Skipped 1 previous similar message [ 5452.497038] Lustre: server umount lustre-OST0000 complete [ 5455.329059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5455.331652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.335582] Lustre: Skipped 1 previous similar message [ 5455.337851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5456.353096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5456.353810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.473510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5461.476329] Lustre: Skipped 2 previous similar messages [ 5466.593497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5466.595704] Lustre: Skipped 1 previous similar message [ 5468.128143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5468.182529] Lustre: server umount lustre-MDT0000 complete [ 5469.303872] LustreError: 218987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502179 with bad export cookie 1786677343479772087 [ 5469.305957] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5469.306791] LustreError: 218987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5469.423441] Lustre: server umount lustre-MDT0001 complete [ 5474.746530] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5478.195940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5478.343855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5479.643990] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5482.519815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.654405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5483.910415] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5484.808937] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5486.827497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5488.595811] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5491.397083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5491.451122] Lustre: lustre-OST0001: new disk, initializing [ 5491.452401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5491.478478] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5491.480317] Lustre: Skipped 1 previous similar message [ 5493.286919] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5493.887720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5493.891536] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5493.914336] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5498.544899] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5504.101801] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5507.852760] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:36:55 (1777502215) [ 5508.576918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5508.580540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.584161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5514.653297] Lustre: server umount lustre-MDT0000 complete [ 5514.722921] LustreError: 222721:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5514.727315] LustreError: 222721:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 5515.875247] LustreError: 222702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502225 with bad export cookie 1786677343479772955 [ 5515.879159] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5515.879705] LustreError: 222702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5515.992147] Lustre: server umount lustre-MDT0001 complete [ 5528.063518] Lustre: server umount lustre-OST0000 complete [ 5538.528824] Lustre: server umount lustre-OST0001 complete [ 5540.710132] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5544.341902] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5556.203897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5556.287477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5556.299362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5556.331948] Lustre: lustre-MDT0000: new disk, initializing [ 5556.354474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5556.361021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5557.638242] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5561.661311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.700203] Lustre: 228863:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5561.793839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5563.076261] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5566.087074] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5567.238522] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5569.434911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5569.528152] Lustre: lustre-OST0000: new disk, initializing [ 5569.529647] Lustre: Skipped 1 previous similar message [ 5569.531548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5569.533974] Lustre: Skipped 2 previous similar messages [ 5571.064864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5571.068655] Lustre: Skipped 1 previous similar message [ 5571.071083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5571.099871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5571.339663] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5574.349858] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5575.658977] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5575.721385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5576.949504] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5577.014452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5581.282527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5581.284924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.285732] LustreError: Skipped 1 previous similar message [ 5581.290362] Lustre: Skipped 3 previous similar messages [ 5581.292853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5581.296043] Lustre: Skipped 5 previous similar messages [ 5583.763303] Lustre: server umount lustre-OST0000 complete [ 5591.124736] Lustre: server umount lustre-MDT0000 complete [ 5591.520786] LustreError: 228873:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5591.525339] LustreError: 228873:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5592.273391] LustreError: 228855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502302 with bad export cookie 1786677343479775048 [ 5592.276244] LustreError: 228855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5592.276431] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5592.377033] Lustre: server umount lustre-MDT0001 complete [ 5595.179527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5595.324106] LustreError: 232668:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.347166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5595.350024] Lustre: Skipped 2 previous similar messages [ 5596.589828] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5599.560845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5601.110064] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5602.914910] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5603.979300] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5606.101911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5607.965166] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5609.682794] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5753.825722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.826759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5753.827828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5753.827833] Lustre: Skipped 2 previous similar messages [ 5753.829470] Lustre: Skipped 3 previous similar messages [ 5759.376592] Lustre: server umount lustre-OST0000 complete [ 5763.552810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5766.808696] Lustre: server umount lustre-MDT0000 complete [ 5768.121650] LustreError: 232650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502477 with bad export cookie 1786677343479775930 [ 5768.126294] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5768.127689] LustreError: 232650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5768.262696] Lustre: server umount lustre-MDT0001 complete [ 5874.730469] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5875.782067] Key type lgssc unregistered [ 5875.917474] LNet: 235918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5875.920575] LNetError: 235918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5875.928313] LNet: Removed LNI 192.168.203.133@tcp [ 5876.262108] Key type .llcrypt unregistered [ 5876.263571] Key type ._llcrypt unregistered [ 5884.631423] Key type ._llcrypt registered [ 5884.632281] Key type .llcrypt registered [ 5884.668663] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5890.524240] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5890.903223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5890.915994] alg: No test for adler32 (adler32-zlib) [ 5891.759814] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 5891.833584] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5893.408150] Key type lgssc registered [ 5893.767435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5907.761726] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5910.801928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5910.807067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.896046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5911.928676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5911.964222] Lustre: lustre-MDT0000: new disk, initializing [ 5911.989393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.994773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5913.192039] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5917.100260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5917.138900] Lustre: 239909:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5917.163493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5917.165097] Lustre: Skipped 1 previous similar message [ 5917.204230] Lustre: lustre-MDT0001: new disk, initializing [ 5917.237208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5917.249995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5917.253160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5918.463176] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5920.646675] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5922.706702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5922.808476] Lustre: lustre-OST0000: new disk, initializing [ 5922.810719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5922.842724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.608788] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5925.622864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5925.626813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5925.661370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5928.628458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5928.695817] Lustre: lustre-OST0001: new disk, initializing [ 5928.697301] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5928.729565] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5930.489476] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5935.614244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5935.617675] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5935.649672] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5935.782413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5939.283335] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5942.752803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5942.755706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5942.759475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5945.826021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.828604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5945.831504] Lustre: Skipped 2 previous similar messages [ 5945.834048] Lustre: Skipped 2 previous similar messages [ 5948.204220] Lustre: server umount lustre-MDT0000 complete [ 5949.346933] LustreError: 239901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502659 with bad export cookie 6096813775086201687 [ 5949.349406] LustreError: 239901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5949.349874] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5949.457924] Lustre: server umount lustre-MDT0001 complete [ 5961.157609] Lustre: server umount lustre-OST0000 complete [ 5972.583198] Lustre: server umount lustre-OST0001 complete [ 5977.253436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5980.060446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5982.586602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5985.526928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5991.182829] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5994.283456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5994.293059] Lustre: lustre-MDT0000: reset Object Index mappings [ 5994.462266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5995.759421] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5998.538751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.544594] Lustre: lustre-MDT0001: reset Object Index mappings [ 6000.022521] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6001.057075] Lustre: 248430:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6003.159593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6003.273242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6003.275531] Lustre: Skipped 1 previous similar message [ 6004.325591] LustreError: 248783:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6004.328543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6004.330438] LustreError: 248783:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6005.044348] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6007.690478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6008.813720] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6009.719231] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6012.879439] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6014.079428] Lustre: 250264:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6019.046076] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.050085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6019.050197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6019.050452] Lustre: Skipped 3 previous similar messages [ 6019.053107] Lustre: Skipped 2 previous similar messages [ 6021.401726] Lustre: server umount lustre-MDT0000 complete [ 6022.687981] LustreError: 250266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502732 with bad export cookie 6096813775086208421 [ 6022.692843] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6022.694046] LustreError: 250266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6022.822075] Lustre: server umount lustre-MDT0001 complete [ 6033.347986] Lustre: server umount lustre-OST0000 complete [ 6044.828186] Lustre: server umount lustre-OST0001 complete [ 6050.539070] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:45:58 (1777502758) [ 6055.249690] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6058.108240] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6069.135160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.224669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6069.235637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6069.269289] Lustre: lustre-MDT0000: new disk, initializing [ 6069.290389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.292102] Lustre: Skipped 1 previous similar message [ 6069.296605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6070.505344] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6074.479956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6074.519891] Lustre: 254476:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6074.632477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6075.799639] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6078.687346] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6079.759506] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6081.918434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6082.015648] Lustre: lustre-OST0000: new disk, initializing [ 6082.016947] Lustre: Skipped 1 previous similar message [ 6082.019087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6082.021128] Lustre: Skipped 2 previous similar messages [ 6083.579983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6083.583449] Lustre: Skipped 1 previous similar message [ 6083.585424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6083.621061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6083.949364] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6086.961226] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6088.264194] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6088.345258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6089.650259] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6089.728148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6093.795055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6093.795674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6093.797160] LustreError: Skipped 1 previous similar message [ 6093.797725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6093.803945] Lustre: Skipped 1 previous similar message [ 6093.808030] Lustre: Skipped 1 previous similar message [ 6096.467710] Lustre: server umount lustre-OST0000 complete [ 6098.913763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.914378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6098.916959] Lustre: Skipped 1 previous similar message [ 6098.919784] Lustre: Skipped 1 previous similar message [ 6103.767322] Lustre: server umount lustre-MDT0000 complete [ 6104.033032] LustreError: 256489:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.038699] LustreError: 256489:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6104.985037] LustreError: 254469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502814 with bad export cookie 6096813775086210507 [ 6104.987615] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6104.990542] LustreError: 254469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6105.115961] Lustre: server umount lustre-MDT0001 complete [ 6111.131347] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6114.575433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.717599] LustreError: 258983:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.733672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.736165] Lustre: Skipped 2 previous similar messages [ 6116.130677] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6119.135878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6120.484722] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6121.393523] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6123.481059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6125.331762] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6128.003800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6128.064131] Lustre: lustre-OST0001: new disk, initializing [ 6128.066083] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6129.471333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6129.474117] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6129.506202] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6129.863755] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6134.258951] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6135.381275] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6137.171923] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6137.179234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6139.875265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6139.875650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6139.878951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6139.881705] Lustre: Skipped 2 previous similar messages [ 6139.884309] Lustre: Skipped 1 previous similar message [ 6144.927414] Lustre: server umount lustre-MDT0000 complete [ 6144.994899] LustreError: 258979:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.000625] LustreError: 258979:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 6146.149145] LustreError: 258964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502855 with bad export cookie 6096813775086211389 [ 6146.152278] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6146.152411] LustreError: 258964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6146.268099] Lustre: server umount lustre-MDT0001 complete [ 6157.322397] Lustre: server umount lustre-OST0000 complete [ 6168.801324] Lustre: server umount lustre-OST0001 complete [ 6173.932394] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6177.181565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.300950] LustreError: 264254:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.437440] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6180.964743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6181.077381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6181.080823] Lustre: Skipped 4 previous similar messages [ 6182.235254] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6183.091275] Lustre: 265362:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6185.109359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6186.878726] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6187.299520] LustreError: 265715:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6187.305310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6189.488888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6190.576910] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6191.411818] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6194.358810] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6195.456826] Lustre: 267201:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6197.193760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6197.203505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6200.263260] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:48:28 (1777502908) [ 6212.064716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6212.067607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6212.070474] Lustre: Skipped 1 previous similar message [ 6212.072052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6217.756459] Lustre: server umount lustre-MDT0000 complete [ 6218.946047] LustreError: 267203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777502928 with bad export cookie 6096813775086217696 [ 6218.948615] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6218.949520] LustreError: 267203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6221.280886] LustreError: 264249:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6221.286051] LustreError: 264249:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6225.135485] Lustre: server umount lustre-MDT0001 complete [ 6232.449320] Lustre: server umount lustre-OST0000 complete [ 6239.831830] Lustre: server umount lustre-OST0001 complete [ 6241.991719] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6245.294879] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6260.156114] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6263.765273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.851358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6263.861705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6263.897092] Lustre: lustre-MDT0000: new disk, initializing [ 6263.924247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6265.158996] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6269.174429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.212267] Lustre: 271382:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6269.320298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6270.559492] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6274.855608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6275.136054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6276.807149] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6280.873882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6282.823259] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6282.936186] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6287.244411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6292.615079] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:50:00 (1777503000) [ 6293.475419] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6293.476865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6293.481805] Lustre: Skipped 7 previous similar messages [ 6293.484477] Lustre: Skipped 9 previous similar messages [ 6299.423873] Lustre: server umount lustre-MDT0000 complete [ 6300.611149] LustreError: 272978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503010 with bad export cookie 6096813775086225578 [ 6300.614292] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6300.614406] LustreError: 272978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6300.733152] Lustre: server umount lustre-MDT0001 complete [ 6311.951465] Lustre: server umount lustre-OST0000 complete [ 6323.544784] Lustre: server umount lustre-OST0001 complete [ 6326.192997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6326.327090] LustreError: 276055:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6326.332593] LustreError: 276055:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6326.350632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6326.352397] Lustre: Skipped 6 previous similar messages [ 6327.517832] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6330.132867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.516689] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6333.215354] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6334.363340] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6336.443415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6338.449847] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6340.190630] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6346.193782] loop1: detected capacity change from 0 to 204800000 [ 6346.257982] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6346.261374] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6346.264819] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6347.899305] loop1: detected capacity change from 0 to 204800000 [ 6349.747792] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6349.780159] LustreError: 276044:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs22760: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6349.786302] LustreError: 276044:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs22760 [ 6349.788990] LustreError: 276044:0:(mgs_handler.c:517:mgs_target_reg()) fs22760-MDT0001: Error checking target index: rc = -13 [ 6349.800797] LustreError: fs22760-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6349.804572] LustreError: 279476:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 6349.807653] LustreError: 279476:0:(tgt_mount.c:2084:server_put_super()) no obd fs22760-MDT0001 [ 6349.810234] LustreError: 279476:0:(tgt_mount.c:116:server_deregister_mount()) fs22760-MDT0001 not registered [ 6349.823390] Lustre: server umount fs22760-MDT0001 complete [ 6349.825263] LustreError: 279476:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6352.614119] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.644620] Lustre: Setting parameter fs22760-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22760-MDT0001 [ 6352.648400] Lustre: Skipped 2 previous similar messages [ 6352.664217] Lustre: srv-fs22760-MDT0001: No data found on store. Initialize space. [ 6352.666081] Lustre: Skipped 4 previous similar messages [ 6352.704729] Lustre: fs22760-MDT0001: new disk, initializing [ 6352.706333] Lustre: Skipped 3 previous similar messages [ 6352.740365] Lustre: srv-fs22760-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6352.743681] Lustre: 279941:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22760-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6352.748196] LustreError: 279941:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs22760-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6354.026668] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6356.780634] Lustre: server umount fs22760-MDT0001 complete [ 6360.803369] loop1: detected capacity change from 0 to 204800000 [ 6362.694558] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6362.796121] Lustre: 281489:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22760-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6362.799738] LustreError: 281489:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs22760-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6364.138154] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6365.763123] Lustre: server umount fs22760-MDT0001 complete [ 6370.277573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6370.278873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6370.281193] Lustre: Skipped 2 previous similar messages [ 6370.285408] Lustre: Skipped 6 previous similar messages [ 6375.070814] Lustre: server umount lustre-MDT0000 complete [ 6377.707976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6377.750061] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6379.091683] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6381.428473] loop1: detected capacity change from 0 to 204800000 [ 6383.077913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6383.088937] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6383.192470] Lustre: 283897:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22760-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6383.195224] LustreError: 283897:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs22760-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6384.500493] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6386.092350] Lustre: server umount fs22760-MDT0001 complete [ 6391.414174] LustreError: 278166:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6391.418873] LustreError: 278166:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 36 previous similar messages [ 6393.320318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6398.666468] Lustre: server umount lustre-OST0000 complete [ 6414.304082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6414.365515] Lustre: server umount lustre-MDT0000 complete [ 6415.496039] LustreError: 276034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503125 with bad export cookie 6096813775086228252 [ 6415.498860] LustreError: 276034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6415.499140] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6415.600122] Lustre: server umount lustre-MDT0001 complete [ 6418.839594] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6419.757848] Key type lgssc unregistered [ 6419.878273] LNet: 286204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6419.880214] LNetError: 286204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6419.888244] LNet: Removed LNI 192.168.203.133@tcp [ 6420.151133] Key type .llcrypt unregistered [ 6420.152047] Key type ._llcrypt unregistered [ 6427.256374] Key type ._llcrypt registered [ 6427.257635] Key type .llcrypt registered [ 6427.290983] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6427.570303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6427.579484] alg: No test for adler32 (adler32-zlib) [ 6428.415672] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6428.486438] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6430.056159] Key type lgssc registered [ 6430.387930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6432.802409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6432.809192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6433.959784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.160722] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6437.819979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6437.935126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6439.138924] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6440.703535] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6441.736749] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6443.800789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6443.927266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6444.964542] LustreError: 289217:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6444.971315] LustreError: 289217:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6445.767116] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6447.413208] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6448.038459] LustreError: 289037:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.045667] LustreError: 289037:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6453.223412] LustreError: 289037:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6453.229121] LustreError: 289037:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6454.372871] loop1: detected capacity change from 0 to 204800000 [ 6454.436940] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6454.440464] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6454.445579] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6456.098239] loop1: detected capacity change from 0 to 204800000 [ 6457.848745] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6457.892784] Lustre: fs512-OST0000: new disk, initializing [ 6457.894510] Lustre: srv-fs512-OST0000: No data found on store. Initialize space. [ 6457.915078] Lustre: fs512-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6458.338382] LustreError: 289218:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6458.342752] LustreError: 289218:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6459.650152] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6462.964826] Lustre: server umount fs512-OST0000 complete [ 6463.458571] LustreError: 289218:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.465934] LustreError: 289218:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6466.895026] loop1: detected capacity change from 0 to 204800000 [ 6468.761297] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6468.811721] Lustre: fs512-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6470.551774] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6473.139831] loop2: detected capacity change from 0 to 204800000 [ 6473.259911] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6473.263642] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6473.267365] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6473.700145] LustreError: 289217:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6473.704725] LustreError: 289217:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 6474.854237] loop2: detected capacity change from 0 to 204800000 [ 6476.594780] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6476.619581] LustreError: fs512-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6476.623918] LustreError: fs512-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6476.627454] LustreError: 293878:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 6476.630701] LustreError: 293878:0:(tgt_mount.c:2084:server_put_super()) no obd fs512-OST0001 [ 6476.633385] LustreError: 293878:0:(tgt_mount.c:116:server_deregister_mount()) fs512-OST0001 not registered [ 6476.643153] Lustre: server umount fs512-OST0001 complete [ 6476.644361] LustreError: 293878:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6479.823840] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6479.862544] Lustre: fs512-OST0001: new disk, initializing [ 6479.864348] Lustre: srv-fs512-OST0001: No data found on store. Initialize space. [ 6479.884610] Lustre: fs512-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6481.672070] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6484.935906] Lustre: server umount fs512-OST0000 complete [ 6487.789969] Lustre: server umount fs512-OST0001 complete [ 6489.718366] LustreError: 289037:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6489.724140] LustreError: 289037:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [ 6491.569370] loop1: detected capacity change from 0 to 204800000 [ 6493.322179] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6495.081996] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6497.053392] loop2: detected capacity change from 0 to 204800000 [ 6498.786589] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6498.847122] Lustre: fs512-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6498.849643] Lustre: Skipped 1 previous similar message [ 6500.520845] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6503.250459] Lustre: server umount fs512-OST0000 complete [ 6506.149199] Lustre: server umount fs512-OST0001 complete [ 6511.584857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6511.587129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6511.591145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6514.662658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6514.662974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6514.664993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6515.798368] Lustre: server umount lustre-OST0000 complete [ 6519.778529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6519.779175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6519.786128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6519.788242] Lustre: Skipped 2 previous similar messages [ 6523.158533] Lustre: server umount lustre-MDT0000 complete [ 6524.322385] LustreError: 287415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503234 with bad export cookie 3928436988878314837 [ 6524.325094] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6524.325229] LustreError: 287415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6524.427541] Lustre: server umount lustre-MDT0001 complete [ 6527.793230] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6528.713570] Key type lgssc unregistered [ 6528.828251] LNet: 300270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6528.830897] LNetError: 300270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6528.841233] LNet: Removed LNI 192.168.203.133@tcp [ 6529.119104] Key type .llcrypt unregistered [ 6529.120224] Key type ._llcrypt unregistered [ 6538.646291] Key type ._llcrypt registered [ 6538.647133] Key type .llcrypt registered [ 6538.684157] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6544.832118] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6545.321338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6545.332345] alg: No test for adler32 (adler32-zlib) [ 6546.187016] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6546.284386] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6547.872156] Key type lgssc registered [ 6548.230970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6559.043520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6559.049175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6560.149501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6560.159562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6560.197867] Lustre: lustre-MDT0000: new disk, initializing [ 6560.220051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.226091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6561.400465] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6565.439329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6565.481552] Lustre: 304064:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6565.506481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6565.508104] Lustre: Skipped 1 previous similar message [ 6565.544212] Lustre: lustre-MDT0001: new disk, initializing [ 6565.576057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6565.590028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6565.595572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6566.959859] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6570.114250] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6571.386486] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6573.781342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6573.917715] Lustre: lustre-OST0000: new disk, initializing [ 6573.920270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6573.955801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6576.038733] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6577.909981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6577.913571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6577.945397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6579.086979] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6579.733177] Lustre: Mounted lustre-client [ 6580.018200] LustreError: 306556:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8b6ecba6a000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 6580.044122] Lustre: Unmounted lustre-client [ 6583.265361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6583.265570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6583.268332] LustreError: Skipped 1 previous similar message [ 6583.272588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6583.275735] Lustre: Skipped 1 previous similar message [ 6584.803901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6584.805528] Lustre: Skipped 1 previous similar message [ 6584.912850] Lustre: server umount lustre-OST0000 complete [ 6586.336710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6586.338947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6586.342272] Lustre: Skipped 1 previous similar message [ 6586.343975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.921221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.921828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6592.154283] Lustre: server umount lustre-MDT0000 complete [ 6593.399594] LustreError: 305805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503303 with bad export cookie 6009060352530721367 [ 6593.402936] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6593.403476] LustreError: 305805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6593.530569] Lustre: server umount lustre-MDT0001 complete [ 6597.152695] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6598.166860] Key type lgssc unregistered [ 6639.264273] LNet: 307832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6639.266983] LNetError: 307832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6710.950309] LNet: Removed LNI 192.168.203.133@tcp [ 6711.244961] Key type .llcrypt unregistered [ 6711.246033] Key type ._llcrypt unregistered [ 6717.141132] Key type ._llcrypt registered [ 6717.142207] Key type .llcrypt registered [ 6717.183990] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6723.468617] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6723.814472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6723.853769] alg: No test for adler32 (adler32-zlib) [ 6724.699794] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6724.778751] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6726.360151] Key type lgssc registered [ 6726.726793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6736.565293] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:57:24 (1777503444) [ 6738.747772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6738.753058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.842777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6739.856300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6739.896040] Lustre: lustre-MDT0000: new disk, initializing [ 6739.921118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6739.928460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6741.301193] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6745.469240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6745.518899] Lustre: 310996:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6745.547344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6745.549366] Lustre: Skipped 1 previous similar message [ 6745.585883] Lustre: lustre-MDT0001: new disk, initializing [ 6745.619102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6745.630682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6745.634308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6746.973419] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6750.061255] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6751.315523] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6753.586568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6753.707148] Lustre: lustre-OST0000: new disk, initializing [ 6753.708802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6753.740356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6755.684930] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6758.802042] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6758.913877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6758.918909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6758.960218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6764.004804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6764.004971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6764.008096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6764.013350] Lustre: Skipped 1 previous similar message [ 6768.783692] Lustre: server umount lustre-OST0000 complete [ 6771.168871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6771.171260] LustreError: Skipped 1 previous similar message [ 6771.172919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6771.178206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6771.180029] Lustre: Skipped 1 previous similar message [ 6774.241392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6774.241957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.156484] Lustre: server umount lustre-MDT0000 complete [ 6777.422382] LustreError: 310988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503487 with bad export cookie 13054021874878541941 [ 6777.426451] LustreError: 310988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6777.426624] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6777.551378] Lustre: server umount lustre-MDT0001 complete [ 6781.228071] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6782.360321] Key type lgssc unregistered [ 6782.504371] LNet: 314365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6782.506854] LNetError: 314365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6782.512444] LNet: Removed LNI 192.168.203.133@tcp [ 6782.819400] Key type .llcrypt unregistered [ 6782.820825] Key type ._llcrypt unregistered [ 6792.523119] Key type ._llcrypt registered [ 6792.524508] Key type .llcrypt registered [ 6792.558631] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6792.917760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6792.963402] alg: No test for adler32 (adler32-zlib) [ 6793.815243] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6793.915179] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6795.496148] Key type lgssc registered [ 6795.913943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6798.536504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6798.541945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6799.699158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6801.014717] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6803.812385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.937704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6805.217693] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6806.998803] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6808.194427] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6810.438276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.557585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6812.501135] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6814.294258] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6832.374634] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6832.945439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6834.656905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6834.659297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6834.663823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6835.169902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6835.170422] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6835.172199] Lustre: Skipped 1 previous similar message [ 6835.175825] Lustre: Skipped 1 previous similar message [ 6836.433725] Lustre: server umount lustre-MDT0000 complete [ 6837.673333] LustreError: 317359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503547 with bad export cookie 4646309799374075303 [ 6837.676443] LustreError: 317359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6837.676963] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6839.926081] LustreError: 315741:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6839.931154] LustreError: 315741:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6839.934079] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.33@tcp (stopping) [ 6839.936140] Lustre: Skipped 1 previous similar message [ 6840.289337] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.336893] LustreError: 315740:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6842.337091] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6842.341408] LustreError: 315740:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6842.345198] Lustre: Skipped 1 previous similar message [ 6843.824356] Lustre: server umount lustre-MDT0001 complete [ 6850.171753] Lustre: lustre-OST0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 6851.150236] Lustre: server umount lustre-OST0000 complete [ 6853.991680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6855.292236] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6856.203186] Lustre: 319491:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.133@tcp [ 6856.883899] Lustre: server umount MGS complete [ 6859.663698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6859.810505] LustreError: 320062:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6859.833319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.099415] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6863.925565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6864.053596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6865.378196] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6868.179372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6870.113312] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6871.893573] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6872.439172] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6873.620411] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6874.158538] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6878.691263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6878.691475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.696358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6878.699101] Lustre: Skipped 1 previous similar message [ 6878.707134] Lustre: Skipped 1 previous similar message [ 6882.892263] Lustre: server umount lustre-OST0000 complete [ 6884.832803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6884.835268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6890.260690] Lustre: server umount lustre-MDT0000 complete [ 6891.495038] LustreError: 320042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503601 with bad export cookie 4646309799374076129 [ 6891.497563] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6891.500246] LustreError: 320042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6891.602510] Lustre: server umount lustre-MDT0001 complete [ 6895.041803] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6896.054038] Key type lgssc unregistered [ 6896.175502] LNet: 323234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6896.178487] LNetError: 323234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6896.188363] LNet: Removed LNI 192.168.203.133@tcp [ 6896.483262] Key type .llcrypt unregistered [ 6896.484475] Key type ._llcrypt unregistered [ 6902.195474] Key type ._llcrypt registered [ 6902.196851] Key type .llcrypt registered [ 6902.238484] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6908.659664] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6908.921813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6908.955995] alg: No test for adler32 (adler32-zlib) [ 6909.811376] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6909.892278] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6911.472229] Key type lgssc registered [ 6911.813252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6923.601480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6923.610713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.708765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6924.720523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6924.755551] Lustre: lustre-MDT0000: new disk, initializing [ 6924.779244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6924.786809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6926.039049] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6930.280211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.319652] Lustre: 326334:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6930.342042] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6930.344199] Lustre: Skipped 1 previous similar message [ 6930.388587] Lustre: lustre-MDT0001: new disk, initializing [ 6930.425857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6930.436750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6930.440540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6931.749912] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6934.807195] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6935.963625] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6938.185159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6938.287949] Lustre: lustre-OST0000: new disk, initializing [ 6938.290856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6938.320271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6940.208814] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6943.316985] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6944.247711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6944.251211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6944.279047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6944.698778] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6944.773401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6946.103195] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6946.183798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6949.347485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6949.347773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6949.349459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6949.349829] LustreError: Skipped 1 previous similar message [ 6949.354040] Lustre: Skipped 1 previous similar message [ 6952.911305] Lustre: server umount lustre-OST0000 complete [ 6954.465332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6954.465957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6954.470021] Lustre: Skipped 1 previous similar message [ 6954.471676] Lustre: Skipped 1 previous similar message [ 6959.585223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6959.587960] Lustre: Skipped 2 previous similar messages [ 6960.280676] Lustre: server umount lustre-MDT0000 complete [ 6961.522177] LustreError: 326324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503671 with bad export cookie 7532594572939823059 [ 6961.525182] LustreError: 326324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6961.525619] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6961.640809] Lustre: server umount lustre-MDT0001 complete [ 6964.530182] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:01:12 (1777503672) [ 6969.742698] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6972.744033] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6984.257575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.349373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6984.362513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6984.400510] Lustre: lustre-MDT0000: new disk, initializing [ 6984.427348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.436870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6985.801942] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6989.864056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6989.902174] Lustre: 332652:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6989.922474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6989.924288] Lustre: Skipped 1 previous similar message [ 6989.959570] Lustre: lustre-MDT0001: new disk, initializing [ 6989.988692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6990.001732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6990.004883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6991.298917] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6994.401529] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6995.640991] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6998.025458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6998.152373] Lustre: lustre-OST0000: new disk, initializing [ 6998.154281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6998.190500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7000.193026] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7002.171057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7002.174584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7002.215180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7003.304582] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7012.833345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7012.836281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7012.841439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7014.737215] Lustre: server umount lustre-OST0000 complete [ 7017.441596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7017.442363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7017.448036] Lustre: Skipped 1 previous similar message [ 7017.454115] Lustre: Skipped 1 previous similar message [ 7027.681164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7027.683061] Lustre: Skipped 3 previous similar messages [ 7030.240126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7030.311168] Lustre: server umount lustre-MDT0000 complete [ 7031.561180] LustreError: 333429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777503741 with bad export cookie 7532594572939823934 [ 7031.566323] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7031.566318] LustreError: 333429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7042.074655] Lustre: server umount lustre-MDT0001 complete [ 7045.572399] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7046.584875] Key type lgssc unregistered [ 7046.715421] LNet: 336415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7046.717700] LNetError: 336415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7046.725358] LNet: Removed LNI 192.168.203.133@tcp [ 7047.029389] Key type .llcrypt unregistered [ 7047.030905] Key type ._llcrypt unregistered [ 7078.281154] Key type ._llcrypt registered [ 7078.282125] Key type .llcrypt registered [ 7078.326398] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7078.434174] Key type .llcrypt unregistered [ 7078.435348] Key type ._llcrypt unregistered [ 7109.662768] Key type ._llcrypt registered [ 7109.665168] Key type .llcrypt registered [ 7109.706627] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7109.824261] Key type .llcrypt unregistered [ 7109.825921] Key type ._llcrypt unregistered [ 7121.005781] Key type ._llcrypt registered [ 7121.006916] Key type .llcrypt registered [ 7121.047541] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7121.139282] Key type .llcrypt unregistered [ 7121.140836] Key type ._llcrypt unregistered [ 7132.291692] Key type ._llcrypt registered [ 7132.292673] Key type .llcrypt registered [ 7132.327854] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7132.429445] Key type .llcrypt unregistered [ 7132.430489] Key type ._llcrypt unregistered [ 7159.597479] Key type ._llcrypt registered [ 7159.598683] Key type .llcrypt registered [ 7159.635214] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7159.734441] Key type .llcrypt unregistered [ 7159.736189] Key type ._llcrypt unregistered [ 7174.865775] Key type ._llcrypt registered [ 7174.867757] Key type .llcrypt registered [ 7174.929044] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7175.090308] Key type .llcrypt unregistered [ 7175.091821] Key type ._llcrypt unregistered [ 7199.502375] Key type ._llcrypt registered [ 7199.508764] Key type .llcrypt registered [ 7199.637760] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7199.958299] Key type .llcrypt unregistered [ 7199.960348] Key type ._llcrypt unregistered [ 7228.074916] Key type ._llcrypt registered [ 7228.078915] Key type .llcrypt registered [ 7228.136552] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7228.278689] Key type .llcrypt unregistered [ 7228.281082] Key type ._llcrypt unregistered [ 7244.104870] Key type ._llcrypt registered [ 7244.107634] Key type .llcrypt registered [ 7244.164317] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7244.357376] Key type .llcrypt unregistered [ 7244.362121] Key type ._llcrypt unregistered [ 7258.729428] Key type ._llcrypt registered [ 7258.731159] Key type .llcrypt registered [ 7258.778338] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7258.899433] Key type .llcrypt unregistered [ 7258.901528] Key type ._llcrypt unregistered [ 7292.436718] Key type ._llcrypt registered [ 7292.440910] Key type .llcrypt registered [ 7292.497119] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7292.642247] Key type .llcrypt unregistered [ 7292.645128] Key type ._llcrypt unregistered [ 7307.791175] Key type ._llcrypt registered [ 7307.792936] Key type .llcrypt registered [ 7307.843322] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7307.974669] Key type .llcrypt unregistered [ 7307.976641] Key type ._llcrypt unregistered [ 7322.541341] Key type ._llcrypt registered [ 7322.543710] Key type .llcrypt registered [ 7322.614327] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7322.789325] Key type .llcrypt unregistered [ 7322.791328] Key type ._llcrypt unregistered [ 7349.893784] Key type ._llcrypt registered [ 7349.895086] Key type .llcrypt registered [ 7349.942450] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 7357.714749] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 7358.098849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7358.119083] alg: No test for adler32 (adler32-zlib) [ 7359.010098] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7359.136564] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 7360.736344] Key type lgssc registered [ 7361.218895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7377.347574] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7377.354848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.492539] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7378.512664] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7378.559304] Lustre: tst250fs-MDT0000: new disk, initializing [ 7378.630405] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7378.645324] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7380.541087] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7386.526802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7386.592842] Lustre: 356107:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7386.634547] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7386.637020] Lustre: Skipped 1 previous similar message [ 7386.688767] Lustre: tst250fs-MDT0001: new disk, initializing [ 7386.740522] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7386.760829] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7386.766948] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7388.522493] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7391.488322] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7394.933275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7395.090398] Lustre: tst250fs-OST0000: new disk, initializing [ 7395.093236] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7395.147725] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7398.249888] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7400.991158] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7401.000818] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7401.074365] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7403.850461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7403.963275] Lustre: tst250fs-OST0001: new disk, initializing [ 7403.965326] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7404.022116] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7406.856410] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7408.661736] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7408.668170] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7408.707343] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7410.239734] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7423.970561] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7423.973802] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7423.980272] Lustre: Skipped 2 previous similar messages [ 7423.987503] Lustre: Skipped 3 previous similar messages [ 7429.091692] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7429.100143] Lustre: Skipped 3 previous similar messages [ 7429.423601] Lustre: server umount tst250fs-MDT0000 complete [ 7431.186703] LustreError: 356885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777504140 with bad export cookie 12347495899134084803 [ 7431.189294] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7431.192487] LustreError: 356885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7431.363569] Lustre: server umount tst250fs-MDT0001 complete [ 7443.483425] Lustre: server umount tst250fs-OST0000 complete [ 7455.609828] Lustre: server umount tst250fs-OST0001 complete [ 7464.254659] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 7467.946530] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 7482.774100] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:09:50 (1777504190) [ 7483.526882] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7490.873826] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 7495.012528] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 7509.751734] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7172 sec ========== 19:10:17 (1777504217) [ 7510.672183] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:10:18 (1777504218) === [ 7512.458798] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:10:20 (1777504220) === [ 7531.097362] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7532.584966] Key type lgssc unregistered [ 7532.758534] LNet: 367571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7532.763195] LNetError: 367571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7532.772695] LNet: Removed LNI 192.168.203.133@tcp [ 7533.172291] Key type .llcrypt unregistered [ 7533.175169] Key type ._llcrypt unregistered