[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 507466142 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003264] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008446] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011147] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013038] SELinux: Initializing. [ 0.014059] *** VALIDATE selinux *** [ 0.022310] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026715] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] *** VALIDATE tmpfs *** [ 0.031037] *** VALIDATE proc *** [ 0.032229] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.035024] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038036] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042342] debug: unmapping init [mem 0xffffffff9aa59000-0xffffffff9aa60fff] [ 0.045200] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046759] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050019] ... value mask: 0000ffffffffffff [ 0.051035] ... max period: 00007fffffffffff [ 0.052022] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054376] rcu: Hierarchical SRCU implementation. [ 0.056741] smp: Bringing up secondary CPUs ... [ 0.057624] x86: Booting SMP configuration: [ 0.058024] .... node #0, CPUs: #1 #2 #3 [ 0.061065] smp: Brought up 1 node, 4 CPUs [ 0.063011] smpboot: Max logical packages: 1 [ 0.064017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145741] node 0 deferred pages initialised in 79ms [ 0.148444] devtmpfs: initialized [ 0.149237] x86/mm: Memory block size: 128MB [ 0.151715] gcov: version magic: 0x41383552 [ 0.153329] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159319] pinctrl core: initialized pinctrl subsystem [ 0.161139] [ 0.161817] ************************************************************* [ 0.164012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166009] ** ** [ 0.168014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171013] ** ** [ 0.173010] ** This means that this kernel is built to expose internal ** [ 0.174009] ** IOMMU data structures, which may compromise security on ** [ 0.175000] ** your system. ** [ 0.175000] ** ** [ 0.175000] ** If you see this message and you are not debugging the ** [ 0.175000] ** kernel, report this immediately to your vendor! ** [ 0.175000] ** ** [ 0.175000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175000] ************************************************************* [ 0.175000] NET: Registered protocol family 16 [ 0.175000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175000] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175000] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.184151] cpuidle: using governor menu [ 0.185000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185000] PCI: Using configuration type 1 for base access [ 0.191179] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.200604] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.204208] cryptd: max_cpu_qlen set to 1000 [ 0.205368] ACPI: Added _OSI(Module Device) [ 0.207011] ACPI: Added _OSI(Processor Device) [ 0.208012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215997] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.223878] ACPI: Interpreter enabled [ 0.225057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.226011] ACPI: Using IOAPIC for interrupt routing [ 0.228103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.232550] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245909] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.251019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259842] acpiphp: Slot [2] registered [ 0.261138] acpiphp: Slot [5] registered [ 0.263160] acpiphp: Slot [6] registered [ 0.264115] acpiphp: Slot [7] registered [ 0.266215] acpiphp: Slot [8] registered [ 0.267144] acpiphp: Slot [9] registered [ 0.269130] acpiphp: Slot [10] registered [ 0.270215] acpiphp: Slot [3] registered [ 0.272088] acpiphp: Slot [4] registered [ 0.274101] acpiphp: Slot [11] registered [ 0.276108] acpiphp: Slot [12] registered [ 0.277110] acpiphp: Slot [13] registered [ 0.279085] acpiphp: Slot [14] registered [ 0.280130] acpiphp: Slot [15] registered [ 0.282145] acpiphp: Slot [16] registered [ 0.283077] acpiphp: Slot [17] registered [ 0.285120] acpiphp: Slot [18] registered [ 0.286128] acpiphp: Slot [19] registered [ 0.287136] acpiphp: Slot [20] registered [ 0.289080] acpiphp: Slot [21] registered [ 0.290087] acpiphp: Slot [22] registered [ 0.292122] acpiphp: Slot [23] registered [ 0.293176] acpiphp: Slot [24] registered [ 0.295214] acpiphp: Slot [25] registered [ 0.296169] acpiphp: Slot [26] registered [ 0.298154] acpiphp: Slot [27] registered [ 0.299191] acpiphp: Slot [28] registered [ 0.301113] acpiphp: Slot [29] registered [ 0.303087] acpiphp: Slot [30] registered [ 0.304131] acpiphp: Slot [31] registered [ 0.306114] PCI host bridge to bus 0000:00 [ 0.307016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317047] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.319026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.321314] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325168] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.328526] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340894] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.347761] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.350024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.355019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.358057] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.360976] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.364093] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366985] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.389015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.396016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402554] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.412015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.423029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.436025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.905356] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.914020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.923019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.950020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.975847] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.990022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.004016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.387027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.399624] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.406023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.411024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.431026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.441807] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 2.056036] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 2.077013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 2.117040] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 2.151090] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 2.229022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 2.236023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 2.251026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 2.264136] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 2.266486] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 2.270442] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 2.273427] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 2.275284] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 2.281083] iommu: Default domain type: Passthrough [ 2.283541] SCSI subsystem initialized [ 2.285189] ACPI: bus type USB registered [ 2.286086] usbcore: registered new interface driver usbfs [ 2.287000] usbcore: registered new interface driver hub [ 2.287000] usbcore: registered new device driver usb [ 2.287000] pps_core: LinuxPPS API ver. 1 registered [ 2.287000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 2.287000] PTP clock support registered [ 2.290166] EDAC MC: Ver: 3.0.0 [ 2.295382] PCI: Using ACPI for IRQ routing [ 2.304008] NetLabel: Initializing [ 2.305008] NetLabel: domain hash size = 128 [ 2.306010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 2.308152] NetLabel: unlabeled traffic allowed by default [ 2.311200] vgaarb: loaded [ 2.313302] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 2.315012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 2.321242] clocksource: Switched to clocksource kvm-clock [ 2.448936] VFS: Disk quotas dquot_6.6.0 [ 2.450869] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.453716] *** VALIDATE ramfs *** [ 2.455221] *** VALIDATE hugetlbfs *** [ 2.456906] pnp: PnP ACPI init [ 2.459705] pnp: PnP ACPI: found 6 devices [ 2.476439] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 2.479984] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 2.482462] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 2.484828] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 2.487376] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 2.490160] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 2.493444] NET: Registered protocol family 2 [ 2.496227] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 2.501601] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 2.504509] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 2.509959] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 2.513780] TCP: Hash tables configured (established 65536 bind 65536) [ 2.516439] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 2.518939] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.521414] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.523980] NET: Registered protocol family 1 [ 2.526173] RPC: Registered named UNIX socket transport module. [ 2.528083] RPC: Registered udp transport module. [ 2.529468] RPC: Registered tcp transport module. [ 2.530842] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.532926] NET: Registered protocol family 44 [ 2.534209] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.535883] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.537606] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.539427] PCI: CLS 0 bytes, default 64 [ 2.541476] Unpacking initramfs... [ 4.186921] debug: unmapping init [mem 0xffff9eeefcc54000-0xffff9eeefffbffff] [ 4.191526] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.193675] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.196894] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.788764] Initialise system trusted keyrings [ 4.790559] Key type blacklist registered [ 4.792472] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.802437] zbud: loaded [ 4.805646] *** VALIDATE nfs *** [ 4.807135] *** VALIDATE nfs4 *** [ 4.808757] pstore: using deflate compression [ 4.812855] Platform Keyring initialized [ 4.934227] NET: Registered protocol family 38 [ 4.935961] Key type asymmetric registered [ 4.937793] Asymmetric key parser 'x509' registered [ 4.940276] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.943832] io scheduler mq-deadline registered [ 4.945740] io scheduler kyber registered [ 4.947680] io scheduler bfq registered [ 4.949727] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.953127] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.956510] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.959955] ACPI: Power Button [PWRF] [ 4.965578] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.974682] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.000496] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.010764] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.044296] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.075961] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.104625] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.109668] Non-volatile memory driver v1.3 [ 5.111592] Linux agpgart interface v0.103 [ 5.147440] virtio_blk virtio1: [vda] 144816 512-byte logical blocks (74.1 MB/70.7 MiB) [ 5.150565] vda: detected capacity change from 0 to 74145792 [ 5.166981] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.170445] vdb: detected capacity change from 0 to 1073741824 [ 5.194354] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.197563] vdc: detected capacity change from 0 to 2621440000 [ 5.215937] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.219645] vdd: detected capacity change from 0 to 2621440000 [ 5.236899] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.240452] vde: detected capacity change from 0 to 4294967296 [ 5.255498] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.258330] vdf: detected capacity change from 0 to 4294967296 [ 5.264616] libphy: Fixed MDIO Bus: probed [ 5.338772] usbcore: registered new interface driver usbserial_generic [ 5.348290] usbserial: USB Serial support registered for generic [ 5.350451] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.355586] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.357806] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.381289] mousedev: PS/2 mouse device common for all mice [ 5.389756] rtc_cmos 00:05: RTC can wake from S4 [ 5.398689] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.401441] rtc_cmos 00:05: registered as rtc0 [ 5.406429] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.412340] intel_pstate: CPU model not supported [ 5.417990] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.419584] hid: raw HID events driver (C) Jiri Kosina [ 5.427452] usbcore: registered new interface driver usbhid [ 5.430449] usbhid: USB HID core driver [ 5.433847] drop_monitor: Initializing network drop monitor service [ 5.436389] Initializing XFRM netlink socket [ 5.456282] NET: Registered protocol family 10 [ 5.462621] Segment Routing with IPv6 [ 5.464853] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.469935] NET: Registered protocol family 17 [ 5.472900] mpls_gso: MPLS GSO support [ 5.484934] RAS: Correctable Errors collector initialized. [ 5.487662] AVX version of gcm_enc/dec engaged. [ 5.489791] AES CTR mode by8 optimization enabled [ 5.910947] sched_clock: Marking stable (5910927285, 0)->(7159089062, -1248161777) [ 5.914304] registered taskstats version 1 [ 5.916323] Loading compiled-in X.509 certificates [ 5.918681] zswap: loaded using pool lzo/zbud [ 5.957703] Key type big_key registered [ 5.980444] Key type encrypted registered [ 5.982446] ima: No TPM chip found, activating TPM-bypass! [ 5.984818] ima: Allocated hash algorithm: sha1 [ 6.030853] ima: No architecture policies found [ 6.033339] evm: Initialising EVM extended attributes: [ 6.035522] evm: security.selinux [ 6.039402] evm: security.ima [ 6.040709] evm: security.capability [ 6.042316] evm: HMAC attrs: 0x1 [ 6.045419] rtc_cmos 00:05: setting system clock to 2026-06-26 13:26:58 UTC (1782480418) [ 6.054651] debug: unmapping init [mem 0xffffffff9ba03000-0xffffffff9bbfffff] [ 6.058794] debug: unmapping init [mem 0xffffffff9a782000-0xffffffff9aa58fff] [ 6.071394] Write protecting the kernel read-only data: 28672k [ 6.075392] debug: unmapping init [mem 0xffffffff98e03000-0xffffffff98ffffff] [ 6.078387] debug: unmapping init [mem 0xffffffff99714000-0xffffffff997fffff] [ 6.132927] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.142710] systemd[1]: Detected virtualization kvm. [ 6.144970] systemd[1]: Detected architecture x86-64. [ 6.147186] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.182585] systemd[1]: No hostname configured. [ 6.184601] systemd[1]: Set hostname to . [ 6.186840] random: systemd: uninitialized urandom read (16 bytes read) [ 6.190325] systemd[1]: Initializing machine ID from random generator. [ 6.291565] random: ln: uninitialized urandom read (6 bytes read) [ 6.637576] random: systemd: uninitialized urandom read (16 bytes read) [ 6.663692] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.674876] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.681920] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.763374] device-mapper: uevent: version 1.0.3 [ 7.765724] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 10.934676] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.154502] random: fast init done Starting dracut initqueue hook... [ 19.417186] scsi host0: ata_piix [ 20.115638] scsi host1: ata_piix [ 20.128748] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 20.154205] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 22.602107] random: crng init done [ 22.616615] random: 7 urandom warning(s) missed due to ratelimiting [ 24.009297] dracut-initqueue[568]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 31.713203] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 36.342887] printk: systemd: 26 output lines suppressed due to ratelimiting [ 37.789397] SELinux: Disabled at runtime. [ 37.863830] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 37.874221] systemd[1]: Detected virtualization kvm. [ 37.876623] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 38.945208] systemd[1]: initrd-switch-root.service: Succeeded. [ 38.948843] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 38.959436] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 38.964549] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 38.968605] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 38.978082] systemd[1]: Starting Journal Service... Starting Journal Service... [ 38.987438] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 40.943908] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 44.843746] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Kernel Device Manager. [ 48.853640] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 49.063318] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [* ] A start job is running for Configur…only root support (12s / no limit) [** ] A start job is running for Configur…only root support (13s / no limit)[ 53.813281] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [*** ] A start job is running for Configur…only root support (15s / no limit)[ 54.055151] EDAC sbridge: Ver: 1.1.2 [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ ***] A start job is running for Configur…only root support (16s / no limit) [ **] A start job is running for Configur…only root support (16s / no limit) [ *] A start job is running for Configur…only root support (17s / no limit) [ **] A start job is running for Configur…only root support (17s / no limit) [ ***] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit) [*** ] A start job is running for Configur…only root support (19s / no limit) [** ] A start job is running for Configur…only root support (20s / no limit) [* ] A start job is running for Configur…only root support (20s / no limit) [** ] A start job is running for Configur…only root support (21s / no limit) [*** ] A start job is running for Configur…only root support (21s / no limit) [ *** ] A start job is running for Configur…only root support (22s / no limit) [ *** ] A start job is running for Configur…only root support (22s / no limit) [ ***] A start job is running for Configur…only root support (23s / no limit)[ 62.766595] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (23s / no limit) [ *] A start job is running for Configur…only root support (24s / no limit) [ **] A start job is running for Configur…only root support (24s / no limit) [ ***] A start job is running for Configur…only root support (25s / no limit) [ *** ] A start job is running for Configur…only root support (25s / no limit)[ 64.998212] NFS: Registering the id_resolver key type [ 65.005542] Key type id_resolver registered [ 65.007059] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (26s / no limit) [*** ] A start job is running for Configur…only root support (26s / no limit) [** ] A start job is running for Configur…only root support (27s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg136-server login: [ 107.124411] hrtimer: interrupt took 3402782 ns [ 146.386500] libcfs: loading out-of-tree module taints kernel. [ 146.533420] Key type ._llcrypt registered [ 146.534716] Key type .llcrypt registered [ 146.734796] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 165.684362] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 167.345411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 167.355395] alg: No test for adler32 (adler32-zlib) [ 168.880175] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 169.816085] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 171.639478] Key type lgssc registered [ 173.727615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 195.956078] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 245.181185] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 258.461583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 258.503771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 259.976835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 260.087871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 260.262890] Lustre: lustre-MDT0000: new disk, initializing [ 260.423257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.451101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 264.942303] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 278.745882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 278.829358] Lustre: 6498:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 278.854582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 278.860024] Lustre: Skipped 1 previous similar message [ 278.930139] Lustre: lustre-MDT0001: new disk, initializing [ 279.020830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 279.059664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 279.074576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 283.964211] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.596923] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 300.168467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 300.508103] Lustre: lustre-OST0000: new disk, initializing [ 300.513907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 300.523707] Lustre: 8437:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 300.580778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 307.224318] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 308.768927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 308.777970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 308.900260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 321.979823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 322.208966] Lustre: lustre-OST0001: new disk, initializing [ 322.211838] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 322.217108] Lustre: 9510:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 322.297306] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 330.281800] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 331.334845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 331.345627] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 331.499253] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 343.831733] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 350.555606] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 357.647910] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing check_logdir /tmp/testlogs/ [ 364.086332] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing yml_node [ 370.267240] Lustre: DEBUG MARKER: Client: 2.17.54.98 [ 373.444551] Lustre: DEBUG MARKER: MDS: 2.17.54.98 [ 376.264987] Lustre: DEBUG MARKER: OSS: 2.17.54.98 [ 378.709931] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 26 09:33:08 EDT 2026 [ 398.886627] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 400.741845] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 417.249978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 417.267159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 417.294143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 418.786268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 418.786965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 418.810130] Lustre: Skipped 1 previous similar message [ 418.843349] Lustre: Skipped 1 previous similar message [ 424.028645] Lustre: server umount lustre-MDT0000 complete [ 428.798196] LustreError: 6491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782480841 with bad export cookie 12705004424400362482 [ 428.803567] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 428.806483] LustreError: 6491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 429.027194] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.029673] LustreError: 7454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 429.030207] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 429.054661] Lustre: Skipped 2 previous similar messages [ 429.107543] LustreError: 7454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 433.056214] LustreError: 7454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 433.056774] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 433.096717] Lustre: Skipped 2 previous similar messages [ 438.240068] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.242501] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 438.273233] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 438.300837] Lustre: Skipped 1 previous similar message [ 443.360245] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 443.932267] Lustre: server umount lustre-MDT0001 complete [ 448.582831] Lustre: server umount lustre-OST0000 complete [ 453.000736] Lustre: server umount lustre-OST0001 complete [ 462.311417] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 474.373045] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 516.403956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.719989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 516.772529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 516.922945] Lustre: lustre-MDT0000: new disk, initializing [ 517.051800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.082397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 522.232316] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 534.961962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.148642] Lustre: 15066:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 535.248202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 535.249986] Lustre: Skipped 1 previous similar message [ 535.362542] Lustre: lustre-MDT0001: new disk, initializing [ 535.508317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 535.579658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 535.597431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 541.795557] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 552.350838] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 559.382933] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 568.373477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 568.840986] Lustre: lustre-OST0000: new disk, initializing [ 568.844933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 568.850140] Lustre: 17033:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 569.003907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 570.059660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 570.081629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 570.255677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 577.666539] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 589.192321] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 594.296553] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 594.488947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 599.810792] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 600.025550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 606.182866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 606.189619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.200881] LustreError: Skipped 1 previous similar message [ 606.202850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 606.215946] Lustre: Skipped 1 previous similar message [ 606.218512] Lustre: Skipped 2 previous similar messages [ 608.526221] Lustre: server umount lustre-OST0000 complete [ 616.417938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.429062] Lustre: Skipped 1 previous similar message [ 619.420573] Lustre: server umount lustre-MDT0000 complete [ 621.538237] LustreError: 15078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.566953] LustreError: 15078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 623.772979] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481036 with bad export cookie 12705004424400364015 [ 623.775480] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.784798] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 624.260475] Lustre: server umount lustre-MDT0001 complete [ 631.657517] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:37:22 (1782481042) [ 653.082877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.787563] LustreError: 19912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.857737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.699516] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 664.033551] LustreError: 19913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.050428] LustreError: 19913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 667.925970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.703658] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 680.352921] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 686.474904] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 688.899192] Lustre: Failing over lustre-MDT0000 [ 689.231560] Lustre: server umount lustre-MDT0000 complete [ 689.633972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.652908] LustreError: 19908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.689065] LustreError: 19908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 705.952442] Lustre: 3633:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782481102/real 1782481102] req@ffff9eef476d0a80 x1869066358920320/t0(0) o400->MGC192.168.201.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782481118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 705.974546] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 708.806371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.273364] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb0513e35c2266775 [ 716.294254] Lustre: MGC192.168.201.136@tcp: Connection restored to 0@lo (at 0@lo) [ 716.838895] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 721.908456] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 721.936907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 721.960231] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 722.423832] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 732.540712] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 737.253387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.259023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 737.262564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 737.262573] Lustre: Skipped 2 previous similar messages [ 737.277053] Lustre: Skipped 2 previous similar messages [ 751.071881] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 751.292177] Lustre: server umount lustre-MDT0000 complete [ 753.638713] LustreError: 19913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 753.665317] LustreError: 19913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 755.454741] LustreError: 19894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481167 with bad export cookie 12705004424400365429 [ 755.459584] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.463784] LustreError: 19894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 755.671887] Lustre: server umount lustre-MDT0001 complete [ 764.404828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 765.140486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 765.143883] Lustre: Skipped 2 previous similar messages [ 769.910729] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 778.436499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 783.956745] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 791.627501] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 797.481692] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 799.663787] Lustre: Failing over lustre-MDT0000 [ 799.692044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.700684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 799.708759] Lustre: Skipped 7 previous similar messages [ 800.038932] Lustre: server umount lustre-MDT0000 complete [ 819.170650] LustreError: 23249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 819.186413] LustreError: 23249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 819.283519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.516291] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 819.932070] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 823.979061] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 823.991545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 824.002336] Lustre: Skipped 1 previous similar message [ 824.032651] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 824.412056] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 833.387536] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 839.648528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 843.397668] Lustre: server umount lustre-MDT0000 complete [ 847.533850] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481259 with bad export cookie 12705004424400366367 [ 847.554192] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 847.568065] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 848.056958] Lustre: server umount lustre-MDT0001 complete [ 856.567731] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:41:07 (1782481267) [ 871.712846] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 880.921071] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 919.134337] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 929.241638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 929.489500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 929.529451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 929.626684] Lustre: lustre-MDT0000: new disk, initializing [ 929.713277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.718101] Lustre: Skipped 2 previous similar messages [ 929.728821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 933.590630] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 943.352726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.483219] Lustre: 29625:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 943.745979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 947.905505] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 956.603626] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 961.326424] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 967.817124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 968.122853] Lustre: 31594:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 970.114232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 970.161361] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 970.263734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 970.288668] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 974.791882] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 983.689600] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 989.496572] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1006.055838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1006.057546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.064091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1006.064100] Lustre: Skipped 3 previous similar messages [ 1006.064800] LustreError: Skipped 1 previous similar message [ 1006.093931] Lustre: Skipped 4 previous similar messages [ 1010.890169] Lustre: server umount lustre-OST0000 complete [ 1015.268818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1020.222731] Lustre: server umount lustre-MDT0000 complete [ 1021.408870] LustreError: 29636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.424570] LustreError: 29636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1023.351761] LustreError: 29616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481435 with bad export cookie 12705004424400366773 [ 1023.354189] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.359229] LustreError: 29616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1023.703920] Lustre: server umount lustre-MDT0001 complete [ 1033.247707] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1035.800810] Key type lgssc unregistered [ 1036.083206] LNet: 33571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.090282] LNetError: 33571:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.127107] LNet: Removed LNI 192.168.201.136@tcp [ 1036.989163] Key type .llcrypt unregistered [ 1036.992159] Key type ._llcrypt unregistered [ 1056.843147] Key type ._llcrypt registered [ 1056.846304] Key type .llcrypt registered [ 1056.946027] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1057.644898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1057.869726] alg: No test for adler32 (adler32-zlib) [ 1058.937884] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 1059.173742] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1060.823365] Key type lgssc registered [ 1061.860460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.753914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1068.766253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.158625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.897450] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1080.288306] LustreError: 34982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1080.303147] LustreError: 34982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1081.971716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.374822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1086.564180] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1093.358408] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1098.094595] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1104.148709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1104.451856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.501827] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1109.163395] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1115.049853] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1127.459911] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:45:38 (1782481538) [ 1128.773883] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1130.235784] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1135.817887] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:45:47 (1782481547) [ 1143.788848] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:45:55 (1782481555) [ 1144.845066] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1151.952575] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1151.960225] Lustre: Skipped 2 previous similar messages [ 1153.433264] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:46:04 (1782481564) [ 1156.331447] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1156.336749] Lustre: Skipped 3 previous similar messages [ 1170.702758] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1170.707829] Lustre: Skipped 6 previous similar messages [ 1172.828456] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:46:24 (1782481584) [ 1175.008114] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1178.020342] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1185.939434] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:46:37 (1782481597) [ 1187.485624] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1187.489029] Lustre: Skipped 3 previous similar messages [ 1198.507817] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:46:49 (1782481609) [ 1209.824746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1209.835709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.847608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1211.882709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1211.883915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.919571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.374505] Lustre: server umount lustre-OST0000 complete [ 1220.576197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1220.588668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.611720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1220.623655] Lustre: Skipped 1 previous similar message [ 1224.650602] Lustre: server umount lustre-MDT0000 complete [ 1227.235971] LustreError: 34976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1227.254218] LustreError: 34976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1227.820380] LustreError: 34964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481640 with bad export cookie 2268088074073015309 [ 1227.826051] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1227.835655] LustreError: 34964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1228.195523] Lustre: server umount lustre-MDT0001 complete [ 1236.561196] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1238.805630] Key type lgssc unregistered [ 1239.068781] LNet: 42836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1239.074531] LNetError: 42836:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.094280] LNet: Removed LNI 192.168.201.136@tcp [ 1239.905642] Key type .llcrypt unregistered [ 1239.909562] Key type ._llcrypt unregistered [ 1255.203477] Key type ._llcrypt registered [ 1255.205777] Key type .llcrypt registered [ 1255.331654] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 1269.621840] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1271.299462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1271.343090] alg: No test for adler32 (adler32-zlib) [ 1272.498739] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 1272.821261] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1274.511189] Key type lgssc registered [ 1275.677795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1306.146483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1306.155976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.367768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1307.390270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1307.516982] Lustre: lustre-MDT0000: new disk, initializing [ 1307.576068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.582792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1310.999192] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1319.657631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.729609] Lustre: 45991:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1319.777742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1319.779912] Lustre: Skipped 1 previous similar message [ 1319.853619] Lustre: lustre-MDT0001: new disk, initializing [ 1319.950322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1319.975816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1319.984685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1323.277905] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1331.258173] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1336.209577] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1341.939618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1342.171136] Lustre: lustre-OST0000: new disk, initializing [ 1342.174690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1342.179636] Lustre: 47957:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1342.258869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1347.000451] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1352.257489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1352.262941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1352.330495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1354.055653] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1357.374965] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1357.576488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1361.516496] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1361.707308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1368.036638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1368.040927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1368.069993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1368.074653] Lustre: Skipped 1 previous similar message [ 1373.155980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1377.247181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1377.374880] Lustre: server umount lustre-OST0000 complete [ 1381.344133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1381.350905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.355533] Lustre: Skipped 1 previous similar message [ 1381.362148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1381.364692] Lustre: Skipped 1 previous similar message [ 1383.394732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.396966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1383.416643] Lustre: Skipped 1 previous similar message [ 1386.186706] Lustre: server umount lustre-MDT0000 complete [ 1388.765378] LustreError: 47956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481801 with bad export cookie 12761955924275748821 [ 1388.773404] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1388.777967] LustreError: 47956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1389.074292] Lustre: server umount lustre-MDT0001 complete [ 1395.786110] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:50:07 (1782481807) [ 1401.280087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.601029] LustreError: 50417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.613061] LustreError: 50417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1401.671043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.986001] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1406.950580] LustreError: 50418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1411.141216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.415257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1415.566696] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1420.773604] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1424.964898] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1429.684825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1429.883307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.314495] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1438.046499] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1451.855446] LustreError: 52966:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1452.683276] Lustre: Modifying parameter general.timeout=1 in log params [ 1454.122140] Lustre: Modifying parameter general.timeout=3 in log params [ 1454.126886] Lustre: Skipped 1 previous similar message [ 1456.369567] Lustre: Modifying parameter general.timeout=6 in log params [ 1456.377486] Lustre: Skipped 2 previous similar messages [ 1460.553954] Lustre: Modifying parameter general.timeout=12 in log params [ 1460.564237] Lustre: Skipped 5 previous similar messages [ 1468.645483] Lustre: Modifying parameter general.timeout=23 in log params [ 1468.657583] Lustre: Skipped 10 previous similar messages [ 1485.357938] Lustre: Modifying parameter general.timeout=45 in log params [ 1485.361742] Lustre: Skipped 21 previous similar messages [ 1517.695250] Lustre: Modifying parameter general.timeout=85 in log params [ 1517.698365] Lustre: Skipped 39 previous similar messages [ 1560.248680] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:52:51 (1782481971) [ 1562.529986] Lustre: 59894:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1564.382509] Lustre: 60003:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1567.791854] Lustre: 60202:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1567.804678] Lustre: 60202:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 1583.069158] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 in log params [ 1583.074653] Lustre: Skipped 62 previous similar messages [ 1591.287819] Lustre: 61773:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1591.296135] Lustre: 61773:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 1591.310202] LustreError: 61773:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1594.642640] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:53:26 (1782482006) [ 1606.113606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1606.127738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.153310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.519556] Lustre: server umount lustre-MDT0000 complete [ 1611.746553] LustreError: 50413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1611.777324] LustreError: 50413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1612.203153] LustreError: 50399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482024 with bad export cookie 12761955924275749703 [ 1612.204477] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1612.212622] LustreError: 50399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1612.776579] Lustre: server umount lustre-MDT0001 complete [ 1626.029641] Lustre: server umount lustre-OST0000 complete [ 1644.240166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.448402] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1644.648366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.101848] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1654.779642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.855586] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1654.883751] Lustre: 64303:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1658.057570] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1663.021329] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1666.505391] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1670.770421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1670.891941] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1675.315660] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1680.199107] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1681.320505] Lustre: 66870:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1689.056295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1689.061107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.076275] Lustre: Skipped 2 previous similar messages [ 1689.082086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1689.088146] Lustre: Skipped 2 previous similar messages [ 1692.374629] Lustre: server umount lustre-OST0000 complete [ 1696.226047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1696.232670] LustreError: Skipped 1 previous similar message [ 1699.813563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.822860] Lustre: Skipped 2 previous similar messages [ 1701.133978] Lustre: server umount lustre-MDT0000 complete [ 1703.984610] LustreError: 65047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482116 with bad export cookie 12761955924275752601 [ 1703.987668] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1703.997174] LustreError: 65047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1704.312209] Lustre: server umount lustre-MDT0001 complete [ 1712.922925] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1715.056324] Key type lgssc unregistered [ 1715.291929] LNet: 68078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.297154] LNetError: 68078:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.307985] LNet: Removed LNI 192.168.201.136@tcp [ 1716.003262] Key type .llcrypt unregistered [ 1716.006767] Key type ._llcrypt unregistered [ 1733.105170] Key type ._llcrypt registered [ 1733.106990] Key type .llcrypt registered [ 1733.188309] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1734.017584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1734.044019] alg: No test for adler32 (adler32-zlib) [ 1735.077679] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 1735.242284] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1736.881864] Key type lgssc registered [ 1737.714675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1743.132914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1743.149533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.583841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.848543] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1753.608047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.003861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1757.197285] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1761.797738] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1766.050947] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1771.050688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1771.520261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.840209] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1780.905827] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1797.554705] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1805.281647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1805.286052] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1805.296090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1805.306312] Lustre: Skipped 2 previous similar messages [ 1805.317128] Lustre: Skipped 2 previous similar messages [ 1809.917957] Lustre: server umount lustre-MDT0000 complete [ 1810.401265] LustreError: 71847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.418302] LustreError: 71847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1812.855720] LustreError: 69453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482225 with bad export cookie 16681403410902213879 [ 1812.859947] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1812.863184] LustreError: 69453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1813.261537] Lustre: server umount lustre-MDT0001 complete [ 1826.547530] Lustre: server umount lustre-OST0000 complete [ 1847.355905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.549923] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1847.570551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1847.789205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.567880] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1858.912917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1858.986402] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1859.014422] Lustre: 74358:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1859.413856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1862.874713] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1868.025400] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1872.128074] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1877.411995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.542245] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1877.861714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1882.914785] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1888.203805] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1889.443423] Lustre: 76932:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1890.830731] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1895.904437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1895.911633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.921563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1900.631471] Lustre: server umount lustre-OST0000 complete [ 1905.635754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1905.642702] LustreError: Skipped 1 previous similar message [ 1905.646119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.661533] Lustre: Skipped 1 previous similar message [ 1905.671513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.679796] Lustre: Skipped 2 previous similar messages [ 1910.340169] Lustre: server umount lustre-MDT0000 complete [ 1911.785147] LustreError: 75479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1911.802302] LustreError: 75479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1913.388904] LustreError: 76226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482325 with bad export cookie 16681403410902214712 [ 1913.393835] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1913.394767] LustreError: 76226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1913.752144] Lustre: server umount lustre-MDT0001 complete [ 1922.533862] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1924.502949] Key type lgssc unregistered [ 1924.761159] LNet: 78142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1924.766946] LNetError: 78142:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.802960] LNet: Removed LNI 192.168.201.136@tcp [ 1926.473818] Key type .llcrypt unregistered [ 1926.476094] Key type ._llcrypt unregistered [ 1944.283801] Key type ._llcrypt registered [ 1944.286149] Key type .llcrypt registered [ 1944.375612] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1945.054384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1945.137841] alg: No test for adler32 (adler32-zlib) [ 1946.125190] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 1946.303440] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1947.959689] Key type lgssc registered [ 1948.674299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1954.078888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1954.088384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.417383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.493954] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1964.584226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.006137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.192385] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1973.239537] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1977.074884] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1981.512912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1981.917308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.424748] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1992.104858] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2006.857221] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2007.513210] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2008.887801] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2008.891773] Lustre: Skipped 1 previous similar message [ 2011.367286] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2011.373635] Lustre: Skipped 3 previous similar messages [ 2015.574525] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2015.577809] Lustre: Skipped 6 previous similar messages [ 2023.816183] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2023.821322] Lustre: Skipped 11 previous similar messages [ 2039.832520] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2039.836492] Lustre: Skipped 22 previous similar messages [ 2071.907580] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2071.909937] Lustre: Skipped 48 previous similar messages [ 2078.804394] Lustre: 87657:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2080.672927] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:01:32 (1782482492) [ 2081.780724] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2083.085441] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:01:34 (1782482494) [ 2087.392293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2087.404360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.412032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2088.416454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.419296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2088.439410] Lustre: Skipped 1 previous similar message [ 2093.539925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2093.542391] Lustre: Skipped 1 previous similar message [ 2098.656978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2098.662381] Lustre: Skipped 1 previous similar message [ 2100.191609] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2100.349854] Lustre: server umount lustre-OST0000 complete [ 2103.268966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2103.277883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.286964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2109.117818] Lustre: server umount lustre-MDT0000 complete [ 2111.823799] LustreError: 79514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482524 with bad export cookie 8714385075272955272 [ 2111.828586] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.829936] LustreError: 79514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2112.051804] Lustre: server umount lustre-MDT0001 complete [ 2121.660583] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2123.781735] Key type lgssc unregistered [ 2123.995833] LNet: 88905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.999308] LNetError: 88905:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.012613] LNet: Removed LNI 192.168.201.136@tcp [ 2124.753303] Key type .llcrypt unregistered [ 2124.755117] Key type ._llcrypt unregistered [ 2128.809141] Key type ._llcrypt registered [ 2128.810711] Key type .llcrypt registered [ 2128.904397] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2132.538970] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules [ 2133.432061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2133.512420] alg: No test for adler32 (adler32-zlib) [ 2134.530506] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 2134.743391] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2136.383287] Key type lgssc registered [ 2137.268717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2138.276154] LustreError: 90108:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2151.841357] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2158.252140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2160.236122] LustreError: 90108:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2160.281803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.705525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.094132] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2170.689634] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:03:02 (1782482582) [ 2171.882620] LustreError: 91131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.893275] LustreError: 91131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2173.535591] LustreError: 91162:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2173.546446] LustreError: 91162:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2173.551527] LustreError: 91162:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 2175.773560] Lustre: server umount lustre-MDT0000 complete [ 2183.860676] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2185.314953] Key type lgssc unregistered [ 2185.476696] LNet: 92654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.480749] LNetError: 92654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.491839] LNet: Removed LNI 192.168.201.136@tcp [ 2185.957173] Key type .llcrypt unregistered [ 2185.958548] Key type ._llcrypt unregistered [ 2197.605994] Key type ._llcrypt registered [ 2197.608390] Key type .llcrypt registered [ 2197.682258] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2198.459950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2198.474597] alg: No test for adler32 (adler32-zlib) [ 2199.390208] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 2199.538053] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2201.159221] Key type lgssc registered [ 2201.759155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2206.397284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2206.408729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.838774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.902390] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2216.331939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.658183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2219.193969] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2223.548871] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2226.952501] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2231.229367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2231.623184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2235.281490] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2239.584675] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2273.152327] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:04:44 (1782482684) [ 2274.234922] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2275.372657] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:04:46 (1782482686) [ 2277.860730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2277.867402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.878841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2281.957746] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.958433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2281.971756] Lustre: Skipped 1 previous similar message [ 2281.979977] Lustre: Skipped 2 previous similar messages [ 2283.995852] Lustre: server umount lustre-MDT0000 complete [ 2286.397535] LustreError: 93831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482698 with bad export cookie 3847362021548190185 [ 2286.398907] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.403549] LustreError: 93831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2286.650593] Lustre: server umount lustre-MDT0001 complete [ 2299.554858] Lustre: server umount lustre-OST0000 complete [ 2308.477332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.793345] LustreError: 98119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.806745] LustreError: 98119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2308.943097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.574137] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2314.208337] LustreError: 98120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.578203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.049957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2321.868252] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2326.360337] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2330.283635] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2339.750935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2339.871930] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2339.875690] LustreError: 98109:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2339.903406] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2339.910995] LustreError: 100201:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2339.915160] LustreError: 100201:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2339.922310] LustreError: 100201:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2339.996197] Lustre: server umount lustre-OST0000 complete [ 2339.997505] LustreError: 100201:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2345.390024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2345.492746] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2345.495918] LustreError: 98107:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2345.499603] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2345.502608] LustreError: 100663:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2345.506405] LustreError: 100663:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2345.508138] LustreError: 100663:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2345.538283] Lustre: server umount lustre-OST0000 complete [ 2345.540658] LustreError: 100663:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2351.959281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2352.083540] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2352.085339] Lustre: Found index 0 for lustre-OST0000, updating log [ 2352.088335] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2352.138393] Lustre: lustre-OST0000: new disk, initializing [ 2352.141324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2352.146943] Lustre: 101197:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2352.363208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2356.588383] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2359.058696] Lustre: Failing over lustre-OST0000 [ 2359.173853] Lustre: server umount lustre-OST0000 complete [ 2363.617703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.631350] Lustre: Skipped 1 previous similar message [ 2363.631885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2363.639983] Lustre: Skipped 1 previous similar message [ 2368.993728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2368.997982] Lustre: Skipped 1 previous similar message [ 2371.101910] Lustre: server umount lustre-MDT0000 complete [ 2373.808655] LustreError: 98101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482786 with bad export cookie 3847362021548192929 [ 2373.812555] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2373.821352] LustreError: 98101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2373.991396] Lustre: server umount lustre-MDT0001 complete [ 2379.593476] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:06:31 (1782482791) [ 2389.843522] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2396.685503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.945838] LustreError: 103538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.995359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.768408] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2402.278541] LustreError: 103538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.688402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2409.278708] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2411.440977] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2416.485671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2416.904200] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2421.596934] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2422.059922] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2422.080285] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2422.080692] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2422.095015] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2428.508618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2428.558449] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2428.638429] Lustre: lustre-OST0001: new disk, initializing [ 2428.641160] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2428.644284] Lustre: 105858:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2433.366446] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2436.689708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2436.693809] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2436.758547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2438.702857] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2445.362890] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2450.677524] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [ 2452.450902] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.463865] Lustre: Skipped 1 previous similar message [ 2454.691827] Lustre: server umount lustre-MDT0001 complete [ 2455.778342] LustreError: 103533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.662623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.742518] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2460.745493] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2460.773243] Lustre: 103528:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2460.888061] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (not set up) [ 2460.891832] Lustre: Skipped 3 previous similar messages [ 2461.209410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2461.212492] Lustre: Skipped 3 previous similar messages [ 2462.530935] LustreError: 107461:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2462.535684] LustreError: 107461:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2462.540495] LustreError: 103531:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2462.592501] LustreError: 107477:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.136@tcp: cfg command failed: rc = -17 [ 2462.597149] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2462.597149] [ 2463.648516] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2466.299571] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2469.217522] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:08:00 (1782482880) [ 2478.596073] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2490.304967] Lustre: 109467:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2505.075096] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2507.299448] Lustre: 110602:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2573.792279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2573.792424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.794281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.804822] Lustre: Skipped 3 previous similar messages [ 2577.461758] Lustre: server umount lustre-MDT0000 complete [ 2578.914820] LustreError: 105422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.924331] LustreError: 105422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2579.228152] LustreError: 104280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482991 with bad export cookie 3847362021548193860 [ 2579.229604] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2579.233876] LustreError: 104280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2579.379565] Lustre: server umount lustre-MDT0001 complete [ 2591.261840] Lustre: server umount lustre-OST0000 complete [ 2603.633819] Lustre: server umount lustre-OST0001 complete [ 2606.261387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2611.290636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2615.411044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2619.769749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2629.491727] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2634.471280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.486761] Lustre: lustre-MDT0000: reset Object Index mappings [ 2634.709891] LustreError: 114544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.765942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.726899] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2641.206279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.223812] Lustre: lustre-MDT0001: reset Object Index mappings [ 2643.612887] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2645.047273] Lustre: 115691:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2648.291627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.802699] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2651.507940] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2655.703731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.108417] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2660.136081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 2660.140232] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2660.140988] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2663.322731] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2665.004465] Lustre: 117567:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2690.508911] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:11:42 (1782483102) [ 2721.765095] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.770965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2721.779023] Lustre: Skipped 4 previous similar messages [ 2721.785545] Lustre: Skipped 6 previous similar messages [ 2724.938256] Lustre: server umount lustre-MDT0000 complete [ 2726.779730] LustreError: 114525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483139 with bad export cookie 3847362021548240935 [ 2726.784261] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2726.793811] LustreError: 114525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2726.880283] LustreError: 114539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.888593] LustreError: 114539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2741.215135] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2741.385857] Lustre: server umount lustre-MDT0001 complete [ 2749.646182] Lustre: server umount lustre-OST0000 complete [ 2757.629500] Lustre: server umount lustre-OST0001 complete [ 2760.798727] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 2765.473604] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2783.579696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.699570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2783.717219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2783.759417] Lustre: lustre-MDT0000: new disk, initializing [ 2783.799665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2783.806622] Lustre: Skipped 3 previous similar messages [ 2783.815955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2786.131148] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2790.061235] Lustre: server umount lustre-MDT0000 complete [ 2794.713204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.870129] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2794.876358] Lustre: 121827:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2794.883088] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2797.091737] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2799.412285] Lustre: server umount lustre-MDT0000 complete [ 2803.616769] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:13:35 (1782483215) [ 2804.393556] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2805.293778] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:13:37 (1782483217) [ 2814.768649] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2820.633611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.103784] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2827.987869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.052908] Lustre: 123874:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2828.086463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2828.089195] Lustre: Skipped 1 previous similar message [ 2828.130616] Lustre: lustre-MDT0001: new disk, initializing [ 2828.199856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2828.206224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2830.746121] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2834.135540] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2837.723775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2837.899060] Lustre: lustre-OST0000: new disk, initializing [ 2837.901388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2837.904177] Lustre: 125448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2839.775241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2839.782332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2839.864615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2841.515822] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2847.601435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2847.683789] Lustre: lustre-OST0001: new disk, initializing [ 2847.686535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2847.691231] Lustre: 126318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2849.303873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2849.308540] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2849.389912] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2851.063492] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2856.923201] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2858.816949] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3058.851264] Lustre: 132702:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3058.855442] Lustre: 132702:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3058.857762] Lustre: 132702:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3058.862161] Lustre: 132702:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3058.864523] Lustre: 132702:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3058.866615] Lustre: 132702:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3197.220090] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:20:08 (1782483608) [ 3209.334000] Lustre: server umount lustre-MDT0000 complete [ 3210.930394] LustreError: 123866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483623 with bad export cookie 3847362021548300113 [ 3210.935428] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3210.935821] LustreError: 123866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3212.770615] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3212.771810] LustreError: 127093:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3212.772549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3212.772553] Lustre: Skipped 8 previous similar messages [ 3212.785576] Lustre: Skipped 4 previous similar messages [ 3212.801163] LustreError: 127093:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3213.170952] Lustre: server umount lustre-MDT0001 complete [ 3221.392213] Lustre: server umount lustre-OST0000 complete [ 3229.354900] Lustre: server umount lustre-OST0001 complete [ 3232.440582] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 3237.105341] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3253.995696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.110236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3254.129876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3254.172324] Lustre: lustre-MDT0000: new disk, initializing [ 3254.224899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3254.227324] Lustre: Skipped 5 previous similar messages [ 3254.235829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3256.376194] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3262.202540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.264469] Lustre: 135738:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3262.429627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3264.605554] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3269.733765] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3272.689680] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3276.608130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3276.795487] Lustre: 137706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3277.854378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3277.859766] Lustre: Skipped 1 previous similar message [ 3277.862947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3277.960318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3280.184141] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3285.484476] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3290.514642] Lustre: lustre-MDD0000: changelog on [ 3291.934873] Lustre: lustre-MDD0001: changelog on [ 4181.994614] Lustre: lustre-MDD0001: changelog off [ 4183.769414] Lustre: 141963:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4184.692409] Lustre: lustre-MDD0000: changelog off [ 4194.275223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4194.277626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.287201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.287209] Lustre: Skipped 1 previous similar message [ 4194.312012] Lustre: Skipped 2 previous similar messages [ 4195.781482] Lustre: server umount lustre-OST0000 complete [ 4204.519080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4204.534475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4204.539473] Lustre: Skipped 1 previous similar message [ 4204.542429] LustreError: Skipped 1 previous similar message [ 4206.938564] Lustre: server umount lustre-MDT0000 complete [ 4209.633777] LustreError: 137109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4209.643583] LustreError: 137109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4210.481548] LustreError: 135730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484622 with bad export cookie 3847362021549103720 [ 4210.485925] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4210.491752] LustreError: 135730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4210.872375] Lustre: server umount lustre-MDT0001 complete [ 4220.721615] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 4223.822623] Key type lgssc unregistered [ 4224.188722] LNet: 146561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4224.203406] LNetError: 146561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4224.228458] LNet: Removed LNI 192.168.201.136@tcp [ 4225.269171] Key type .llcrypt unregistered [ 4225.270531] Key type ._llcrypt unregistered [ 4245.473192] Key type ._llcrypt registered [ 4245.479693] Key type .llcrypt registered [ 4245.584676] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4262.441330] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4263.658527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4263.798929] alg: No test for adler32 (adler32-zlib) [ 4264.890852] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 4265.229704] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 4266.927939] Key type lgssc registered [ 4267.801549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4300.675613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4300.692585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.098645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4302.140882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4302.203848] Lustre: lustre-MDT0000: new disk, initializing [ 4302.272538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4302.288333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4306.651582] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4316.131732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.249447] Lustre: 149763:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4316.294321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4316.297232] Lustre: Skipped 1 previous similar message [ 4316.367877] Lustre: lustre-MDT0001: new disk, initializing [ 4316.469234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4316.559496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4316.571461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4320.276325] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4327.357491] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4331.849179] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4337.015882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4337.224710] Lustre: lustre-OST0000: new disk, initializing [ 4337.228743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4337.234447] Lustre: 151730:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4337.320886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.157386] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4344.870205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4344.876186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4344.933464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4347.558725] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4352.001701] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_module obdecho/obdecho [ 4352.969621] LustreError: 152733:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4352.974945] LustreError: 152733:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4354.049094] LustreError: 152829:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 4360.164807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.168215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4360.175187] Lustre: Skipped 1 previous similar message [ 4365.282599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4365.286211] Lustre: Skipped 4 previous similar messages [ 4366.016431] Lustre: server umount lustre-MDT0000 complete [ 4368.112281] LustreError: 149755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484780 with bad export cookie 11124695729881420975 [ 4368.116024] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4368.117545] LustreError: 149755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4368.327692] Lustre: server umount lustre-MDT0001 complete [ 4379.627092] Lustre: server umount lustre-OST0000 complete [ 4389.407901] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4393.526339] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4406.844439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4406.935639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4406.949184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4406.986328] Lustre: lustre-MDT0000: new disk, initializing [ 4407.017969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4407.026830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4408.829240] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4413.956171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.006446] Lustre: 156540:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4414.033613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4414.036684] Lustre: Skipped 1 previous similar message [ 4414.076498] Lustre: lustre-MDT0001: new disk, initializing [ 4414.109001] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4414.123279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4414.128220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4415.836234] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4420.459709] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4423.076289] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4426.141437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4426.265857] Lustre: lustre-OST0000: new disk, initializing [ 4426.268103] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4426.271460] Lustre: 158506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4426.312863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.761766] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4432.403179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4432.414641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4432.471906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4433.670111] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4436.103821] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4436.227068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4438.053089] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4438.137918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4442.593047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4442.595214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4442.596422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4442.601044] Lustre: Skipped 2 previous similar messages [ 4447.713829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4447.720746] Lustre: Skipped 2 previous similar messages [ 4452.834406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4452.837131] Lustre: Skipped 1 previous similar message [ 4453.343179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4453.433547] Lustre: server umount lustre-OST0000 complete [ 4455.393591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4455.398864] LustreError: Skipped 1 previous similar message [ 4455.400885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.953190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4461.352702] Lustre: server umount lustre-MDT0000 complete [ 4462.910035] LustreError: 158505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484875 with bad export cookie 11124695729881421934 [ 4462.913122] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4462.917588] LustreError: 158505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4463.049595] Lustre: server umount lustre-MDT0001 complete [ 4464.779487] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:41:16 (1782484876) [ 4465.385675] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4466.150492] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:41:18 (1782484878) [ 4469.411977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4469.673564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4471.282783] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4474.668496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.625262] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4481.522643] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4486.186656] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4494.216434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4501.634959] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4508.208594] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4514.025630] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 4514.029845] Lustre: Skipped 3 previous similar messages [ 4515.301428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.309313] Lustre: Skipped 1 previous similar message [ 4515.808386] Lustre: server umount lustre-MDT0000 complete [ 4519.125325] LustreError: 160913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.154973] LustreError: 160913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4519.932871] LustreError: 160896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484932 with bad export cookie 11124695729881422802 [ 4519.949561] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4519.952553] LustreError: 160896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4520.416667] LustreError: 162753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4524.261162] LustreError: 160911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4526.667229] Lustre: server umount lustre-MDT0001 complete [ 4537.985132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4558.014213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.653724] LustreError: 164756:0:(llog.c:1646:llog_backup()) MGC192.168.201.136@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4558.682222] Lustre: 164756:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.136@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4564.308854] LustreError: 164761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4564.339111] LustreError: 164761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4570.838506] LustreError: 164761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4570.872599] LustreError: 164761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4579.035040] LustreError: 164763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.077218] LustreError: 164763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4595.177168] LustreError: 164763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4595.195764] LustreError: 164763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 4619.743439] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4619.752192] Lustre: 164756:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.136@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4630.228714] LustreError: 164787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4630.258666] LustreError: 164787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 4660.211364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4660.227437] Lustre: Skipped 2 previous similar messages [ 4664.273198] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4665.328888] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4670.957571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4689.891315] LustreError: 147985:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bcad49c7 x1869070652783488/t0(0) o250->MGC192.168.201.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4690.100306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 4693.691141] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4695.530924] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4699.442438] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4700.779087] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4704.840727] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4706.218865] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4721.123136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4721.128402] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.138118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4721.147150] Lustre: Skipped 3 previous similar messages [ 4721.150272] Lustre: Skipped 6 previous similar messages [ 4725.149181] Lustre: server umount lustre-MDT0000 complete [ 4726.243761] LustreError: 164763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4726.251816] LustreError: 164763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 46 previous similar messages [ 4727.656871] LustreError: 162757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485140 with bad export cookie 11124695729881429249 [ 4727.666435] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4727.925576] Lustre: server umount lustre-MDT0001 complete [ 4740.964840] Lustre: server umount lustre-OST0000 complete [ 4759.553873] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4766.776116] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4796.306097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.482708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4796.504800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4796.570574] Lustre: lustre-MDT0000: new disk, initializing [ 4796.627706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4796.630728] Lustre: Skipped 1 previous similar message [ 4796.644627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4800.262023] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4808.862701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4808.962552] Lustre: 169858:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4809.149582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4813.368749] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4822.403621] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4827.957694] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4833.433417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4833.658655] Lustre: lustre-OST0000: new disk, initializing [ 4833.661504] Lustre: Skipped 1 previous similar message [ 4833.664889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4833.667530] Lustre: Skipped 2 previous similar messages [ 4833.669994] Lustre: 171826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4839.656148] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4843.546561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4843.550206] Lustre: Skipped 1 previous similar message [ 4843.553609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4843.648850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4847.448342] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4851.769393] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4851.921344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4856.253624] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4856.434373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4858.850307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4858.852243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4858.864574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4858.872019] Lustre: Skipped 2 previous similar messages [ 4858.882212] Lustre: Skipped 3 previous similar messages [ 4864.270910] Lustre: server umount lustre-OST0000 complete [ 4873.706963] Lustre: server umount lustre-MDT0000 complete [ 4874.210495] LustreError: 169867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4874.220486] LustreError: 169867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4876.880846] LustreError: 169850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485289 with bad export cookie 11124695729881446084 [ 4876.888167] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4876.891455] LustreError: 169850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4877.266380] Lustre: server umount lustre-MDT0001 complete [ 4883.683225] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:48:15 (1782485295) [ 4888.654382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4892.792737] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4898.783100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4902.226160] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4907.414504] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4911.102692] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4916.000908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4921.257494] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4926.807700] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4944.496637] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:49:15 (1782485355) [ 4950.516293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4950.521360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4950.522364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4950.522370] Lustre: Skipped 4 previous similar messages [ 4950.525534] Lustre: Skipped 3 previous similar messages [ 4953.230994] Lustre: server umount lustre-OST0000 complete [ 4962.113100] Lustre: server umount lustre-MDT0000 complete [ 4964.744992] LustreError: 174218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485377 with bad export cookie 11124695729881446966 [ 4964.748651] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4964.751633] LustreError: 174218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4965.032299] Lustre: server umount lustre-MDT0001 complete [ 4973.240807] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 4975.042053] Key type lgssc unregistered [ 4975.249996] LNet: 177987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4975.255880] LNetError: 177987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4975.272135] LNet: Removed LNI 192.168.201.136@tcp [ 4975.875272] Key type .llcrypt unregistered [ 4975.877269] Key type ._llcrypt unregistered [ 4989.636331] Key type ._llcrypt registered [ 4989.638193] Key type .llcrypt registered [ 4989.727715] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4990.391944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4990.451773] alg: No test for adler32 (adler32-zlib) [ 4991.411210] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 4991.614726] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 4993.247181] Key type lgssc registered [ 4994.007923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4998.933227] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4998.947084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5014.431419] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5019.551303] LustreError: 179208:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.136@tcp: failed processing log, type 4: rc = -110 [ 5024.672594] LustreError: 179208:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5024.680967] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5024.687171] LustreError: MGC192.168.201.136@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5024.698322] LustreError: 179208:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5024.711217] LustreError: 179208:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 5024.713939] LustreError: 179208:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5024.753398] Lustre: server umount lustre-OST0000 complete [ 5024.755086] LustreError: 179208:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5030.356872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.646053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5035.033103] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5041.125311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5041.356232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5044.735305] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5050.297514] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5054.067920] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5059.204153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5059.369316] LustreError: 181522:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5059.382523] Lustre: 181522:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.136@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5059.584726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.253674] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5069.313070] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5071.131143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5071.139487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5071.146955] Lustre: Skipped 2 previous similar messages [ 5071.159913] Lustre: Skipped 1 previous similar message [ 5076.453306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5076.460426] Lustre: Skipped 2 previous similar messages [ 5077.461704] Lustre: server umount lustre-MDT0000 complete [ 5079.853830] LustreError: 179690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485492 with bad export cookie 521945208044642231 [ 5079.857705] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5079.860611] LustreError: 179690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5080.087945] Lustre: server umount lustre-MDT0001 complete [ 5093.340234] Lustre: server umount lustre-OST0000 complete [ 5100.170284] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5107.907637] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5133.842722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5134.044760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5134.064048] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5134.126462] Lustre: lustre-MDT0000: new disk, initializing [ 5134.176788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5134.187524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5137.477152] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5145.527403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5145.672572] Lustre: 185045:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5145.724573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5145.729500] Lustre: Skipped 1 previous similar message [ 5145.806233] Lustre: lustre-MDT0001: new disk, initializing [ 5145.879286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5145.906896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5145.918055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5149.266404] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5155.886814] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5159.647956] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5163.885485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5164.027997] Lustre: lustre-OST0000: new disk, initializing [ 5164.030564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5164.035195] Lustre: 187007:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5164.082296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.086926] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5174.316243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5174.324195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5174.391452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5175.085876] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5178.150090] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5178.309133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5181.277751] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5181.408964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5184.481660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5184.483813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.491489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5184.497846] Lustre: Skipped 1 previous similar message [ 5188.465087] Lustre: server umount lustre-OST0000 complete [ 5192.166013] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5192.173656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5192.181431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5192.186461] Lustre: Skipped 1 previous similar message [ 5194.721074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.781087] Lustre: server umount lustre-MDT0000 complete [ 5198.975194] LustreError: 185035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485611 with bad export cookie 521945208044642945 [ 5198.977161] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5198.981487] LustreError: 185035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5199.190622] Lustre: server umount lustre-MDT0001 complete [ 5204.302661] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:53:36 (1782485616) [ 5213.604272] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 5215.187149] Key type lgssc unregistered [ 5215.426899] LNet: 189947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5215.434950] LNetError: 189947:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5215.448168] LNet: Removed LNI 192.168.201.136@tcp [ 5216.152199] Key type .llcrypt unregistered [ 5216.154420] Key type ._llcrypt unregistered [ 5227.314977] Key type ._llcrypt registered [ 5227.316126] Key type .llcrypt registered [ 5227.376269] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5228.108909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5228.123160] alg: No test for adler32 (adler32-zlib) [ 5229.042952] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 5229.193851] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 5230.823211] Key type lgssc registered [ 5231.497314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5235.737976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5235.750251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5237.047752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5239.690863] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5245.006435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5245.223984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5247.751739] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5252.199194] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5255.510941] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5260.179827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5260.288896] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5260.291208] LustreError: 193005:0:(llog.c:1687:llog_backup()) MGC192.168.201.136@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5260.296108] Lustre: 193005:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.136@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5260.356785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5264.463422] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5268.661648] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5271.009287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5271.019082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5271.028166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5275.105438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5275.106392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.109877] Lustre: Skipped 1 previous similar message [ 5275.117183] Lustre: Skipped 1 previous similar message [ 5275.935583] Lustre: server umount lustre-MDT0000 complete [ 5279.696417] LustreError: 191124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485692 with bad export cookie 11495640882804916404 [ 5279.711096] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5280.110975] Lustre: server umount lustre-MDT0001 complete [ 5294.686289] Lustre: server umount lustre-OST0000 complete [ 5311.052455] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5328.135401] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5378.690225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5378.968318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5379.023569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5379.090812] Lustre: lustre-MDT0000: new disk, initializing [ 5379.156497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5379.169504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5384.445388] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5395.991944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5396.159069] Lustre: 196555:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5396.221226] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5396.224975] Lustre: Skipped 1 previous similar message [ 5396.227380] LustreError: 197337:0:(llog.c:1687:llog_backup()) MGC192.168.201.136@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5396.234110] LustreError: 197337:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 5396.238550] Lustre: 197337:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.136@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5396.251727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5396.255059] Lustre: Skipped 1 previous similar message [ 5396.347408] Lustre: lustre-MDT0001: new disk, initializing [ 5396.499709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5396.567155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5396.576597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5401.192295] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5410.429587] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5415.584597] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5422.375828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5422.589951] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5422.592286] Lustre: Skipped 1 previous similar message [ 5422.596390] LustreError: 198498:0:(llog.c:1687:llog_backup()) MGC192.168.201.136@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5422.605202] LustreError: 198498:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 5422.612140] Lustre: 198498:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.136@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5422.621661] Lustre: 198498:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5422.641660] Lustre: lustre-OST0000: new disk, initializing [ 5422.647262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5422.659697] Lustre: 198524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5422.775240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5428.656481] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5428.839143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5428.853145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5428.966756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5438.450418] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5443.496372] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5443.721444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5448.952033] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5449.290494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5454.822459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5454.823903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.827245] LustreError: Skipped 1 previous similar message [ 5454.828177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5454.828183] Lustre: Skipped 1 previous similar message [ 5454.860994] Lustre: Skipped 1 previous similar message [ 5457.889724] Lustre: server umount lustre-OST0000 complete [ 5463.017652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5463.024823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.041128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5463.050682] Lustre: Skipped 1 previous similar message [ 5470.176152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5470.185184] Lustre: Skipped 3 previous similar messages [ 5476.319310] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5476.608795] Lustre: server umount lustre-MDT0000 complete [ 5480.206529] LustreError: 198523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485892 with bad export cookie 11495640882804917125 [ 5480.209055] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5480.224031] LustreError: 198523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5480.693985] Lustre: server umount lustre-MDT0001 complete [ 5489.777726] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:58:20 (1782485900) [ 5508.327371] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5519.753211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5520.262186] LustreError: 201651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.294868] LustreError: 201651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5520.344670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5525.154484] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5525.473271] LustreError: 201652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5529.583468] LustreError: 201651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5533.260276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.854189] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5540.426741] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5547.365818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5553.888885] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5562.299659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5562.396701] Lustre: lustre-OST0001: new disk, initializing [ 5562.399916] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5562.404469] Lustre: 203959:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5562.489216] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5562.497185] Lustre: Skipped 2 previous similar messages [ 5569.648298] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5569.677553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5569.704104] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5569.821537] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5581.588472] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5591.266218] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5593.391031] Lustre: DEBUG MARKER: ADD OST3 [ 5599.921273] loop: module loaded [ 5600.018712] loop0: detected capacity change from 0 to 204800000 [ 5600.404319] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5600.414228] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5600.425542] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5607.722673] loop0: detected capacity change from 0 to 204800000 [ 5608.310652] Lustre: DEBUG MARKER: STOP OST3 [ 5614.803393] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5615.242611] Lustre: lustre-OST0003: new disk, initializing [ 5615.270136] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5615.283283] Lustre: 206003:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5616.364217] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5616.491792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5616.509336] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5616.580711] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 5618.408168] LustreError: 206098:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5618.797342] Lustre: Failing over lustre-OST0003 [ 5618.938037] Lustre: server umount lustre-OST0003 complete [ 5621.741378] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5621.743717] LustreError: 203135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.749438] Lustre: Skipped 2 previous similar messages [ 5621.765518] LustreError: 203135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5623.459740] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5626.859385] LustreError: 203133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5626.869804] LustreError: 203133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5631.411481] Lustre: DEBUG MARKER: START OST3 again [ 5637.092535] LustreError: 203135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5637.118924] LustreError: 203135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5638.813261] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5639.191651] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5639.196906] Lustre: Skipped 1 previous similar message [ 5639.205047] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5640.554972] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5640.660753] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5640.669589] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5646.370482] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5668.254208] Lustre: Failing over lustre-OST0003 [ 5668.380291] Lustre: server umount lustre-OST0003 complete [ 5669.867950] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5669.871314] LustreError: 203133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5669.880771] Lustre: Skipped 1 previous similar message [ 5669.890835] LustreError: 203133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5687.269323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5687.283291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5687.320083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5687.334079] Lustre: Skipped 2 previous similar messages [ 5692.737164] Lustre: server umount lustre-MDT0000 complete [ 5696.183203] LustreError: 201631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486108 with bad export cookie 11495640882804918007 [ 5696.194311] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5696.202941] LustreError: 201631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5702.854418] Lustre: server umount lustre-MDT0001 complete [ 5706.266535] Lustre: server umount lustre-OST0000 complete [ 5709.765976] Lustre: server umount lustre-OST0001 complete [ 5715.640596] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5723.628562] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5751.546781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.730955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5751.778105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5751.869337] Lustre: lustre-MDT0000: new disk, initializing [ 5751.953770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5755.034411] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5762.440943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5762.546139] Lustre: 211435:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5762.737573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5765.531060] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5771.530339] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5775.207494] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5779.465796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5779.640748] Lustre: lustre-OST0000: new disk, initializing [ 5779.643879] Lustre: Skipped 1 previous similar message [ 5779.647899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5779.653174] Lustre: Skipped 2 previous similar messages [ 5779.657988] Lustre: 213398:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5779.716926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.721478] Lustre: Skipped 2 previous similar messages [ 5781.584030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5781.597704] Lustre: Skipped 1 previous similar message [ 5781.616664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5781.690760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5784.159565] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5790.085347] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5792.602757] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5792.726765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5795.288718] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5795.426787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5797.347499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5797.351804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5797.354700] LustreError: Skipped 1 previous similar message [ 5797.360300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5797.367375] Lustre: Skipped 6 previous similar messages [ 5797.380091] Lustre: Skipped 9 previous similar messages [ 5802.825568] Lustre: server umount lustre-OST0000 complete [ 5811.438188] Lustre: server umount lustre-MDT0000 complete [ 5812.705669] LustreError: 211440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5812.717162] LustreError: 211440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 5813.587458] LustreError: 211427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486226 with bad export cookie 11495640882804920303 [ 5813.592933] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5813.594682] LustreError: 211427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5813.769901] Lustre: server umount lustre-MDT0001 complete [ 5815.712629] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:03:47 (1782486227) [ 5824.354141] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5830.403389] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5851.839545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.946366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5851.967836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5852.010724] Lustre: lustre-MDT0000: new disk, initializing [ 5852.065593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5854.637965] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5861.687861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5861.784343] Lustre: 218001:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5861.971694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5864.352125] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5869.722105] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5872.720834] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5876.418901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5876.599979] Lustre: 219969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5877.979869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5879.815197] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5885.712965] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5888.199226] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5888.315229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5890.515934] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5890.631143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5893.608805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5893.611486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.617054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5893.618709] Lustre: Skipped 3 previous similar messages [ 5893.627160] Lustre: Skipped 5 previous similar messages [ 5897.701318] Lustre: server umount lustre-OST0000 complete [ 5905.844739] Lustre: server umount lustre-MDT0000 complete [ 5907.752230] LustreError: 217994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486320 with bad export cookie 11495640882804921185 [ 5907.756436] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5907.757431] LustreError: 217994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5907.951903] Lustre: server umount lustre-MDT0001 complete [ 5912.277442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5912.478102] LustreError: 222182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.529096] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5918.597019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.693364] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5924.974427] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5928.041419] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5931.456140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5934.371960] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5937.660461] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5947.360516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5947.365437] LustreError: Skipped 2 previous similar messages [ 5951.279195] Lustre: server umount lustre-OST0000 complete [ 5959.325314] Lustre: server umount lustre-MDT0000 complete [ 5961.378857] LustreError: 222164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486373 with bad export cookie 11495640882804922067 [ 5961.386799] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5961.580568] Lustre: server umount lustre-MDT0001 complete [ 6073.936571] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 6075.398844] Key type lgssc unregistered [ 6075.571698] LNet: 225840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6075.574869] LNetError: 225840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6075.586634] LNet: Removed LNI 192.168.201.136@tcp [ 6076.094125] Key type .llcrypt unregistered [ 6076.096083] Key type ._llcrypt unregistered [ 6088.388227] Key type ._llcrypt registered [ 6088.389521] Key type .llcrypt registered [ 6088.445693] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 6095.951582] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6096.438478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6096.453288] alg: No test for adler32 (adler32-zlib) [ 6097.364830] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 6097.472762] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 6099.079126] Key type lgssc registered [ 6099.604565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6115.939441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6115.970488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.111753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6117.128643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6117.169167] Lustre: lustre-MDT0000: new disk, initializing [ 6117.215963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6117.235375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6119.698932] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6125.718782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.782308] Lustre: 229166:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6125.829502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6125.831558] Lustre: Skipped 1 previous similar message [ 6125.877607] Lustre: lustre-MDT0001: new disk, initializing [ 6125.939631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6125.961714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6125.970843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6127.946702] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6133.182175] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6135.925608] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6139.255276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6139.414630] Lustre: lustre-OST0000: new disk, initializing [ 6139.419794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6139.423517] Lustre: 231135:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6139.462155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6142.309777] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6142.735409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6142.743912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6142.817107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6147.122643] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6149.538629] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6149.667578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6151.731125] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6151.835710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6153.189362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6153.190527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6153.196278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6153.202791] Lustre: Skipped 1 previous similar message [ 6153.208477] Lustre: Skipped 1 previous similar message [ 6158.315737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6158.324775] Lustre: Skipped 1 previous similar message [ 6158.957315] Lustre: server umount lustre-OST0000 complete [ 6161.887979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6161.891262] LustreError: Skipped 1 previous similar message [ 6161.893186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6161.898780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6163.429155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.546420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6168.550696] Lustre: Skipped 3 previous similar messages [ 6173.665449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6173.672196] Lustre: Skipped 1 previous similar message [ 6175.199145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6175.305749] Lustre: server umount lustre-MDT0000 complete [ 6177.216253] LustreError: 229159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486589 with bad export cookie 5666728560442937740 [ 6177.217130] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6177.234447] LustreError: 229159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6177.374088] Lustre: server umount lustre-MDT0001 complete [ 6185.757353] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6190.390444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6190.609657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6192.457666] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6195.681300] LustreError: 234064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.694252] LustreError: 234064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6196.772493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6196.952229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6198.985516] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6200.398486] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6203.466456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6206.569558] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6210.775580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6210.858294] Lustre: lustre-OST0001: new disk, initializing [ 6210.860983] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6210.868877] Lustre: 236368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6210.933729] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6210.938907] Lustre: Skipped 1 previous similar message [ 6214.185675] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6217.247131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6217.252390] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6217.315634] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6220.717566] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6224.110690] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6230.241514] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:10:41 (1782486641) [ 6232.552088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.555257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6232.557294] Lustre: Skipped 2 previous similar messages [ 6232.562043] Lustre: Skipped 3 previous similar messages [ 6233.476836] Lustre: server umount lustre-MDT0000 complete [ 6235.927800] LustreError: 234045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486648 with bad export cookie 5666728560442938622 [ 6235.933505] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6235.933726] LustreError: 234045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6236.138936] Lustre: server umount lustre-MDT0001 complete [ 6249.704066] Lustre: server umount lustre-OST0000 complete [ 6263.857574] Lustre: server umount lustre-OST0001 complete [ 6272.767929] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 6283.979251] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6335.872608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.208780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6336.265156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6336.378420] Lustre: lustre-MDT0000: new disk, initializing [ 6336.500141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6336.531330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6341.241086] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6351.818186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.966922] Lustre: 240316:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6352.021828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6352.024140] Lustre: Skipped 1 previous similar message [ 6352.097335] Lustre: lustre-MDT0001: new disk, initializing [ 6352.285160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6352.303256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6356.187722] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6364.842522] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6370.143605] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6376.812614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6377.019633] Lustre: lustre-OST0000: new disk, initializing [ 6377.022058] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6377.026336] Lustre: 242283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6377.121325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6377.127089] Lustre: Skipped 1 previous similar message [ 6378.288022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6378.294476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6378.385340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6383.032301] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6391.860312] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6396.267327] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6396.463768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6401.170947] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6401.367678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6404.076435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6404.081343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.094697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6404.109037] Lustre: Skipped 2 previous similar messages [ 6404.128248] Lustre: Skipped 1 previous similar message [ 6417.375142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6417.750669] Lustre: server umount lustre-OST0000 complete [ 6424.033577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6424.042890] LustreError: Skipped 1 previous similar message [ 6424.045150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6427.652102] Lustre: server umount lustre-MDT0000 complete [ 6429.670680] LustreError: 242367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6429.693273] LustreError: 242367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6431.042300] LustreError: 240306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486843 with bad export cookie 5666728560442940757 [ 6431.053415] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6431.428941] Lustre: server umount lustre-MDT0001 complete [ 6439.040297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6439.367171] LustreError: 244502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6443.281952] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6444.515270] LustreError: 244503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.609703] LustreError: 244502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6451.351996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6451.830486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6451.842630] Lustre: Skipped 1 previous similar message [ 6456.131276] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6463.942623] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6469.641129] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6476.113317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6483.091918] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6490.066108] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6636.512590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6636.520813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.531095] Lustre: Skipped 1 previous similar message [ 6636.542706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6636.544367] Lustre: Skipped 6 previous similar messages [ 6642.389429] Lustre: server umount lustre-OST0000 complete [ 6646.240208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6646.249713] LustreError: Skipped 1 previous similar message [ 6651.578282] Lustre: server umount lustre-MDT0000 complete [ 6652.897847] LustreError: 244499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6652.911368] LustreError: 244499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6654.590479] LustreError: 244484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487067 with bad export cookie 5666728560442941639 [ 6654.593898] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6654.600240] LustreError: 244484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6654.906622] Lustre: server umount lustre-MDT0001 complete [ 6752.700720] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 6754.123977] Key type lgssc unregistered [ 6754.290440] LNet: 248109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6754.294783] LNetError: 248109:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6754.309618] LNet: Removed LNI 192.168.201.136@tcp [ 6754.727142] Key type .llcrypt unregistered [ 6754.728684] Key type ._llcrypt unregistered [ 6766.932405] Key type ._llcrypt registered [ 6766.933855] Key type .llcrypt registered [ 6766.988188] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 6774.890609] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6775.380521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6775.417924] alg: No test for adler32 (adler32-zlib) [ 6776.306766] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 6776.439934] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 6778.039147] Key type lgssc registered [ 6778.567169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6797.418118] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6801.261881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6801.267531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6802.352221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6802.363778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6802.400670] Lustre: lustre-MDT0000: new disk, initializing [ 6802.424915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6802.433121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6803.926277] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6808.760256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6808.809754] Lustre: 252149:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6808.847922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6808.850325] Lustre: Skipped 1 previous similar message [ 6808.885994] Lustre: lustre-MDT0001: new disk, initializing [ 6808.922622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6808.938336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6808.941732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6810.416530] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6812.891775] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6815.545245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6815.657176] Lustre: lustre-OST0000: new disk, initializing [ 6815.659474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6815.661956] Lustre: 253782:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6815.695534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6816.043276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6816.046499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6816.106823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6817.765153] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6822.218758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6822.286993] Lustre: lustre-OST0001: new disk, initializing [ 6822.289281] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6822.292608] Lustre: 254653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6822.333375] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6822.593891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6822.597501] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6822.625188] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6824.334916] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6830.055301] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6836.964538] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6843.360935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6843.364688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6843.367516] Lustre: Skipped 3 previous similar messages [ 6843.372304] Lustre: Skipped 3 previous similar messages [ 6846.319987] Lustre: server umount lustre-MDT0000 complete [ 6847.732371] LustreError: 252144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487260 with bad export cookie 3225961456502049723 [ 6847.736048] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6847.740664] LustreError: 252144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6847.861419] Lustre: server umount lustre-MDT0001 complete [ 6859.465153] Lustre: server umount lustre-OST0000 complete [ 6871.029383] Lustre: server umount lustre-OST0001 complete [ 6877.204623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6881.308772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6884.577758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6888.464897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6896.021513] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6900.014514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.022140] Lustre: lustre-MDT0000: reset Object Index mappings [ 6900.196226] LustreError: 259650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6900.204338] LustreError: 259650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6900.220851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6901.683688] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6904.946822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6904.958743] Lustre: lustre-MDT0001: reset Object Index mappings [ 6906.708241] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6907.846150] Lustre: 260792:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6910.438367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6910.588192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6910.590271] Lustre: Skipped 1 previous similar message [ 6913.220629] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6914.729164] LustreError: 261144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.156765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6919.607726] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6924.778785] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6924.782860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6928.185368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6929.512977] Lustre: 262660:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6930.911873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6930.919517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6930.924517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6935.012922] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6935.014302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6935.020686] Lustre: Skipped 2 previous similar messages [ 6935.023580] Lustre: Skipped 2 previous similar messages [ 6936.923120] Lustre: server umount lustre-MDT0000 complete [ 6938.141159] LustreError: 260394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487350 with bad export cookie 3225961456502056485 [ 6938.145139] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6938.145620] LustreError: 260394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6938.257307] Lustre: server umount lustre-MDT0001 complete [ 6950.336561] Lustre: server umount lustre-OST0000 complete [ 6961.883354] Lustre: server umount lustre-OST0001 complete [ 6968.335480] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:23:00 (1782487380) [ 6973.843590] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 6977.350766] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6989.981445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6990.067347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6990.079050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6990.113834] Lustre: lustre-MDT0000: new disk, initializing [ 6990.132608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6990.134203] Lustre: Skipped 1 previous similar message [ 6990.138716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6991.571314] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6996.021279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6996.066777] Lustre: 266910:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6996.181532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6997.628713] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7001.387864] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7003.252770] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7005.894686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.006732] Lustre: lustre-OST0000: new disk, initializing [ 7006.008584] Lustre: Skipped 1 previous similar message [ 7006.010264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7006.012086] Lustre: Skipped 2 previous similar messages [ 7006.014899] Lustre: 268877:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7007.551777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7007.555060] Lustre: Skipped 1 previous similar message [ 7007.557139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7007.586778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7008.127682] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7011.975152] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7013.427198] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7013.503822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7014.945083] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7015.023490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7017.956210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7017.956941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7017.958660] LustreError: Skipped 1 previous similar message [ 7017.963679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7017.967584] Lustre: Skipped 1 previous similar message [ 7021.776410] Lustre: server umount lustre-OST0000 complete [ 7023.072218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.072710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7023.077429] Lustre: Skipped 2 previous similar messages [ 7029.208602] Lustre: server umount lustre-MDT0000 complete [ 7030.476270] LustreError: 266902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487442 with bad export cookie 3225961456502058571 [ 7030.478279] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7030.480948] LustreError: 266902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7030.618359] Lustre: server umount lustre-MDT0001 complete [ 7037.155335] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7041.130561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7041.267060] LustreError: 271805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7041.282597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7041.284828] Lustre: Skipped 2 previous similar messages [ 7042.806187] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7046.156083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7047.822707] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7048.853257] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7051.151970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7053.381408] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7056.426407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7056.493157] Lustre: lustre-OST0001: new disk, initializing [ 7056.494689] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7056.497287] Lustre: 274114:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7057.881765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7057.886350] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7057.927234] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7058.727513] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7063.466338] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7064.828225] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7067.025287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7067.061711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7072.225311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7072.228525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7072.234317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7072.237059] Lustre: Skipped 3 previous similar messages [ 7075.364302] Lustre: server umount lustre-MDT0000 complete [ 7076.641562] LustreError: 271787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487489 with bad export cookie 3225961456502059453 [ 7076.642764] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7076.645600] LustreError: 271787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7076.769636] Lustre: server umount lustre-MDT0001 complete [ 7088.577324] Lustre: server umount lustre-OST0000 complete [ 7100.130661] Lustre: server umount lustre-OST0001 complete [ 7105.616467] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7109.138206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.291106] LustreError: 277160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7109.322540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7109.324946] Lustre: Skipped 3 previous similar messages [ 7110.820360] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7113.880892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7115.498184] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7116.557925] Lustre: 278300:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7118.800435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7119.971672] LustreError: 278652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7119.977210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7120.906692] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7123.718827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7124.846793] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7126.010392] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7129.140159] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7130.323107] Lustre: 280176:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7132.111572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7132.131468] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7135.391575] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:25:47 (1782487547) [ 7150.047820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7150.049988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7150.053507] Lustre: Skipped 3 previous similar messages [ 7150.055773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7150.057750] Lustre: Skipped 4 previous similar messages [ 7152.980762] Lustre: server umount lustre-MDT0000 complete [ 7154.205244] LustreError: 280178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487566 with bad export cookie 3225961456502065746 [ 7154.207420] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7154.210419] LustreError: 280178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7155.679894] LustreError: 277155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.685331] LustreError: 277155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7160.383485] Lustre: server umount lustre-MDT0001 complete [ 7167.743184] Lustre: server umount lustre-OST0000 complete [ 7183.327259] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7183.396645] Lustre: server umount lustre-OST0001 complete [ 7185.500902] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 7188.445609] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7202.492982] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7205.823799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.907605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7205.918695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7205.955485] Lustre: lustre-MDT0000: new disk, initializing [ 7205.980669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7207.413844] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7211.438277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7211.475108] Lustre: 284405:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7211.574762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7212.860729] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7217.295991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7217.399541] Lustre: 286041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7218.337404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7219.287910] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7223.410432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7225.329333] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7225.381663] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7229.696606] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7234.835234] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:27:26 (1782487646) [ 7235.554110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7235.557609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7235.561033] Lustre: Skipped 8 previous similar messages [ 7235.563246] Lustre: Skipped 9 previous similar messages [ 7241.703842] Lustre: server umount lustre-MDT0000 complete [ 7242.949243] LustreError: 284400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487655 with bad export cookie 3225961456502073600 [ 7242.952517] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7242.952761] LustreError: 284400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7243.071587] Lustre: server umount lustre-MDT0001 complete [ 7254.016398] Lustre: server umount lustre-OST0000 complete [ 7265.497248] Lustre: server umount lustre-OST0001 complete [ 7268.281564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7268.443596] LustreError: 289154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7268.450493] LustreError: 289154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7268.474920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7268.477457] Lustre: Skipped 7 previous similar messages [ 7269.800960] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7272.793577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7274.312141] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7276.850159] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7278.598595] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7280.843287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7283.056677] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7285.517368] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7301.332801] LustreError: 290984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7301.338059] LustreError: 290984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7302.178613] loop1: detected capacity change from 0 to 204800000 [ 7302.255114] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7302.259182] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7302.263173] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7303.899631] loop1: detected capacity change from 0 to 204800000 [ 7305.698811] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7305.725366] LustreError: 289143:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs24489: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7305.729127] LustreError: 289143:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs24489 [ 7305.731032] LustreError: 289143:0:(mgs_handler.c:517:mgs_target_reg()) fs24489-MDT0001: Error checking target index: rc = -13 [ 7305.734083] LustreError: fs24489-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7305.737788] LustreError: 292895:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 7305.740486] LustreError: 292895:0:(tgt_mount.c:2129:server_put_super()) no obd fs24489-MDT0001 [ 7305.743099] LustreError: 292895:0:(tgt_mount.c:116:server_deregister_mount()) fs24489-MDT0001 not registered [ 7305.754983] Lustre: server umount fs24489-MDT0001 complete [ 7305.757066] LustreError: 292895:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7308.598964] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7308.630737] Lustre: Setting parameter fs24489-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs24489-MDT0001 [ 7308.635076] Lustre: Skipped 2 previous similar messages [ 7308.652621] Lustre: srv-fs24489-MDT0001: No data found on store. Initialize space. [ 7308.654687] Lustre: Skipped 4 previous similar messages [ 7308.692244] Lustre: fs24489-MDT0001: new disk, initializing [ 7308.693869] Lustre: Skipped 3 previous similar messages [ 7308.729296] Lustre: srv-fs24489-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7308.733167] Lustre: 293360:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24489-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7308.736891] LustreError: 293360:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24489-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7310.097616] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7313.254674] Lustre: server umount fs24489-MDT0001 complete [ 7317.686223] loop1: detected capacity change from 0 to 204800000 [ 7319.667402] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.795461] Lustre: 294925:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24489-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7319.799240] LustreError: 294925:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24489-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7321.151196] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7322.782629] Lustre: server umount fs24489-MDT0001 complete [ 7326.943109] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 7326.945309] Lustre: Skipped 5 previous similar messages [ 7329.251446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7329.258352] Lustre: Skipped 2 previous similar messages [ 7329.261894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7332.152805] Lustre: server umount lustre-MDT0000 complete [ 7336.259496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.327362] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7339.208967] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7341.554764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7345.945102] loop1: detected capacity change from 0 to 204800000 [ 7350.450895] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7350.686043] Lustre: 297368:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24489-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7350.689752] LustreError: 297368:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24489-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7357.182813] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7366.065660] Lustre: server umount fs24489-MDT0001 complete [ 7367.148819] LustreError: 291666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7367.201494] LustreError: 291666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 7372.266749] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7431.686640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7433.905519] Lustre: server umount lustre-OST0000 complete [ 7446.010474] Lustre: server umount lustre-MDT0000 complete [ 7452.670899] LustreError: 289134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487865 with bad export cookie 3225961456502076274 [ 7452.671885] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7452.689413] LustreError: 289134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7453.548990] Lustre: server umount lustre-MDT0001 complete [ 7475.806270] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7479.691502] Key type lgssc unregistered [ 7480.031944] LNet: 299745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7480.037266] LNetError: 299745:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7480.054853] LNet: Removed LNI 192.168.201.136@tcp [ 7481.304182] Key type .llcrypt unregistered [ 7481.306146] Key type ._llcrypt unregistered [ 7504.887200] Key type ._llcrypt registered [ 7504.890655] Key type .llcrypt registered [ 7504.999962] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7506.753099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7506.773033] alg: No test for adler32 (adler32-zlib) [ 7507.839429] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 7508.060911] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 7509.775654] Key type lgssc registered [ 7511.145046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7519.241637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7519.254176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.685983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.736349] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7530.979769] LustreError: 301017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7531.007051] LustreError: 301017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7532.451755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7532.883537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7537.582884] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7544.759445] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7551.173903] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7559.143191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7559.474655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7565.632423] LustreError: 302845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.614209] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7570.924534] LustreError: 302845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7575.385913] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7576.041285] LustreError: 302844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7582.423425] LustreError: 302844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7582.452785] LustreError: 302844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7592.667433] LustreError: 302845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7592.689553] LustreError: 302845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7597.154110] loop1: detected capacity change from 0 to 204800000 [ 7597.451590] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7597.464130] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7597.477416] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7602.785781] loop1: detected capacity change from 0 to 204800000 [ 7608.813354] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7608.816198] LustreError: 302845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7608.854280] LustreError: 302845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 7609.001969] Lustre: fs21534-OST0000: new disk, initializing [ 7609.005332] Lustre: srv-fs21534-OST0000: No data found on store. Initialize space. [ 7609.012768] Lustre: 304752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7609.096337] Lustre: fs21534-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7616.293693] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7624.843499] Lustre: server umount fs21534-OST0000 complete [ 7637.558680] loop1: detected capacity change from 0 to 204800000 [ 7643.522828] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7643.710193] Lustre: fs21534-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7643.870456] LustreError: 303520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7643.899302] LustreError: 303520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 7644.649321] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 7650.225607] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7658.946600] loop2: detected capacity change from 0 to 204800000 [ 7659.159308] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7659.171826] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7659.197113] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7660.008362] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7665.406207] loop2: detected capacity change from 0 to 204800000 [ 7671.481547] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7671.560906] LustreError: fs21534-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7671.567610] LustreError: fs21534-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7671.573339] LustreError: 307791:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 7671.577682] LustreError: 307791:0:(tgt_mount.c:2129:server_put_super()) no obd fs21534-OST0001 [ 7671.581822] LustreError: 307791:0:(tgt_mount.c:116:server_deregister_mount()) fs21534-OST0001 not registered [ 7671.623604] Lustre: server umount fs21534-OST0001 complete [ 7671.626710] LustreError: 307791:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7683.864288] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7684.173613] Lustre: fs21534-OST0001: new disk, initializing [ 7684.186112] Lustre: srv-fs21534-OST0001: No data found on store. Initialize space. [ 7684.201985] Lustre: 308343:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7684.306691] Lustre: fs21534-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7691.779872] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7700.732937] Lustre: server umount fs21534-OST0000 complete [ 7710.452832] LustreError: 302844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7710.486982] LustreError: 302844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [ 7710.581982] Lustre: server umount fs21534-OST0001 complete [ 7724.140022] loop1: detected capacity change from 0 to 204800000 [ 7730.599769] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7730.836219] Lustre: fs21534-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7737.597269] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7744.178243] loop2: detected capacity change from 0 to 204800000 [ 7749.417526] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7755.690829] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7767.017984] Lustre: server umount fs21534-OST0000 complete [ 7779.677088] Lustre: server umount fs21534-OST0001 complete [ 7796.192330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7796.206726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7796.228298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7798.251128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7798.264676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7798.281417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7798.287885] LustreError: Skipped 1 previous similar message [ 7799.951226] Lustre: server umount lustre-OST0000 complete [ 7804.387157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7804.400914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.426690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7804.444677] Lustre: Skipped 1 previous similar message [ 7808.482538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7808.482695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7810.138821] Lustre: server umount lustre-MDT0000 complete [ 7813.655542] LustreError: 300999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488226 with bad export cookie 6140765973646385641 [ 7813.658795] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7813.669268] LustreError: 300999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7814.090558] Lustre: server umount lustre-MDT0001 complete [ 7826.426322] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7828.541356] Key type lgssc unregistered [ 7828.856765] LNet: 314268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7828.864793] LNetError: 314268:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7828.882301] LNet: Removed LNI 192.168.201.136@tcp [ 7829.875172] Key type .llcrypt unregistered [ 7829.878478] Key type ._llcrypt unregistered [ 7868.276468] Key type ._llcrypt registered [ 7868.285226] Key type .llcrypt registered [ 7868.471689] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 7886.862237] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7888.086366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7888.098062] alg: No test for adler32 (adler32-zlib) [ 7889.112731] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 7889.455172] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 7891.193931] Key type lgssc registered [ 7892.054644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7933.274995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7933.325053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7934.733860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7934.777144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7934.896733] Lustre: lustre-MDT0000: new disk, initializing [ 7935.038986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7935.076161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7941.104116] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7953.470361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7953.705548] Lustre: 318134:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7953.778696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7953.783877] Lustre: Skipped 1 previous similar message [ 7953.876274] Lustre: lustre-MDT0001: new disk, initializing [ 7953.996779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7954.042156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7954.058887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7958.409904] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7969.943659] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7977.679319] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7986.917931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.453247] Lustre: lustre-OST0000: new disk, initializing [ 7987.457629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7987.465113] Lustre: 320101:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7987.639670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7992.927834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7992.941480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7993.182086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7996.612662] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8007.111399] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8009.205340] Lustre: Mounted lustre-client [ 8010.257607] Lustre: Unmounted lustre-client [ 8024.044479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8024.055434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.063039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8024.067145] Lustre: Skipped 1 previous similar message [ 8024.087134] Lustre: Skipped 1 previous similar message [ 8029.680216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8029.696858] Lustre: Skipped 1 previous similar message [ 8034.787814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8034.792902] Lustre: Skipped 1 previous similar message [ 8036.319647] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8036.655502] Lustre: server umount lustre-OST0000 complete [ 8045.033215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.042602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8048.901593] Lustre: server umount lustre-MDT0000 complete [ 8052.370667] LustreError: 318125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488464 with bad export cookie 17977039726721053155 [ 8052.383883] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8052.386317] LustreError: 318125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8052.840406] Lustre: server umount lustre-MDT0001 complete [ 8061.121354] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8063.324907] Key type lgssc unregistered [ 8071.071140] LNet: 1 local NIs in recovery (showing 1): 192.168.201.136@tcp [ 8071.711848] LNet: 322265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8071.719227] LNetError: 322265:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8140.325337] LNet: Removed LNI 192.168.201.136@tcp [ 8140.692858] Key type .llcrypt unregistered [ 8140.694237] Key type ._llcrypt unregistered [ 8148.319674] Key type ._llcrypt registered [ 8148.321213] Key type .llcrypt registered [ 8148.377723] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 8155.668419] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8156.156413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8156.228832] alg: No test for adler32 (adler32-zlib) [ 8157.095629] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 8157.190690] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8158.775188] Key type lgssc registered [ 8159.250137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8169.952643] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:43:02 (1782488582) [ 8171.986367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8171.990640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8173.086340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8173.095180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8173.130094] Lustre: lustre-MDT0000: new disk, initializing [ 8173.150994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8173.157590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8174.546112] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8178.608550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8178.651397] Lustre: 325460:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8178.679863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8178.681969] Lustre: Skipped 1 previous similar message [ 8178.715292] Lustre: lustre-MDT0001: new disk, initializing [ 8178.748525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8178.760495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8178.763684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8180.118877] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8183.617425] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8185.292784] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8187.407692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8187.509501] Lustre: lustre-OST0000: new disk, initializing [ 8187.510906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8187.512995] Lustre: 327425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8187.542255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.320682] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8192.683912] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8194.040726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8194.044167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8194.072453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8204.258314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8204.258403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8204.259921] LustreError: Skipped 1 previous similar message [ 8204.260404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8204.264528] Lustre: Skipped 1 previous similar message [ 8208.979093] Lustre: server umount lustre-OST0000 complete [ 8214.496583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8214.496755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8214.497203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8214.497207] Lustre: Skipped 1 previous similar message [ 8214.503448] Lustre: Skipped 1 previous similar message [ 8216.274740] Lustre: server umount lustre-MDT0000 complete [ 8217.474462] LustreError: 325451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488629 with bad export cookie 7996384366845424553 [ 8217.477898] LustreError: 325451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8217.477983] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8217.590720] Lustre: server umount lustre-MDT0001 complete [ 8221.160633] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8222.227428] Key type lgssc unregistered [ 8222.369381] LNet: 329334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8222.371537] LNetError: 329334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8222.379281] LNet: Removed LNI 192.168.201.136@tcp [ 8222.713357] Key type .llcrypt unregistered [ 8222.714527] Key type ._llcrypt unregistered [ 8230.738787] Key type ._llcrypt registered [ 8230.739730] Key type .llcrypt registered [ 8230.775977] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8231.105576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8231.123233] alg: No test for adler32 (adler32-zlib) [ 8231.978392] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 8232.070179] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8233.655166] Key type lgssc registered [ 8234.028263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8236.330726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8236.338236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8237.457814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8238.833620] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8241.557040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8241.691479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8242.988288] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8245.178324] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8246.812917] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8249.059968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8249.170876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8250.907151] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8253.052622] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8271.622970] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8272.110978] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8273.624716] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 8273.889137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8273.893688] Lustre: Skipped 2 previous similar messages [ 8275.478336] Lustre: server umount lustre-MDT0000 complete [ 8276.692143] LustreError: 332425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488689 with bad export cookie 11355063427569049667 [ 8276.695348] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8276.695840] LustreError: 332425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8278.739337] LustreError: 333072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8278.744150] LustreError: 333072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8278.745233] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [ 8278.748409] Lustre: Skipped 3 previous similar messages [ 8279.008960] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8281.055717] LustreError: 330579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8281.055979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8281.059709] LustreError: 330579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8281.061856] Lustre: Skipped 1 previous similar message [ 8282.846443] Lustre: server umount lustre-MDT0001 complete [ 8288.984254] Lustre: lustre-OST0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 8290.129498] Lustre: server umount lustre-OST0000 complete [ 8292.624424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.798746] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8294.712739] Lustre: 334779:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.136@tcp [ 8295.354656] Lustre: server umount MGS complete [ 8298.025579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8298.171052] LustreError: 335348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8298.192375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8299.634895] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8302.516227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8302.641077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8304.066761] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8307.077568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8309.229242] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8311.763208] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8312.358117] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8314.273364] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8314.902645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8318.436583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.437562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8318.438484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8318.440560] Lustre: Skipped 1 previous similar message [ 8318.445961] LustreError: Skipped 1 previous similar message [ 8322.322578] Lustre: server umount lustre-OST0000 complete [ 8328.160895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8328.164271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8328.171202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8328.175260] Lustre: Skipped 1 previous similar message [ 8329.822094] Lustre: server umount lustre-MDT0000 complete [ 8331.382200] LustreError: 335329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488743 with bad export cookie 11355063427569050493 [ 8331.384839] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8331.389357] LustreError: 335329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8331.546740] Lustre: server umount lustre-MDT0001 complete [ 8337.502487] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8339.128405] Key type lgssc unregistered [ 8339.303761] LNet: 338780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8339.308109] LNetError: 338780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8339.319587] LNet: Removed LNI 192.168.201.136@tcp [ 8339.859108] Key type .llcrypt unregistered [ 8339.860457] Key type ._llcrypt unregistered [ 8349.808873] Key type ._llcrypt registered [ 8349.812277] Key type .llcrypt registered [ 8349.900971] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 8359.536936] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8360.086964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8360.174840] alg: No test for adler32 (adler32-zlib) [ 8361.172093] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 8361.333506] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8362.943162] Key type lgssc registered [ 8363.494089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8385.455744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8385.528763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8386.917510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8386.969439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8387.095501] Lustre: lustre-MDT0000: new disk, initializing [ 8387.194090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8387.223414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8392.427076] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8401.366711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8401.497786] Lustre: 341917:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8401.538236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8401.541346] Lustre: Skipped 1 previous similar message [ 8401.638093] Lustre: lustre-MDT0001: new disk, initializing [ 8401.729483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8401.779548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8401.788901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8405.382547] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8411.917847] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8415.872653] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8419.934315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8420.119405] Lustre: lustre-OST0000: new disk, initializing [ 8420.123574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8420.128449] Lustre: 343883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8420.201260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8423.302399] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8427.030073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8427.035461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8427.085612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8428.158569] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8429.941375] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8430.033359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8431.851153] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8431.932467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8434.840585] Lustre: server umount lustre-OST0000 complete [ 8437.218946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.221981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8437.229727] Lustre: Skipped 1 previous similar message [ 8442.336572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8442.340290] Lustre: Skipped 2 previous similar messages [ 8447.457891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8447.460712] Lustre: Skipped 1 previous similar message [ 8447.535642] Lustre: server umount lustre-MDT0000 complete [ 8449.986058] LustreError: 341911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488862 with bad export cookie 11843931444752261783 [ 8449.988813] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8450.000263] LustreError: 341911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8460.383984] Lustre: server umount lustre-MDT0001 complete [ 8466.136901] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:47:57 (1782488877) [ 8475.660365] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 8481.581912] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8501.393588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8501.508937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8501.532843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8501.572194] Lustre: lustre-MDT0000: new disk, initializing [ 8501.606471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8501.614608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8503.659970] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8509.846897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.916948] Lustre: 348639:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8509.948220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8509.952289] Lustre: Skipped 1 previous similar message [ 8509.999667] Lustre: lustre-MDT0001: new disk, initializing [ 8510.064160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8510.088033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8510.093141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8512.889374] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8519.113021] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8522.652791] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8527.337272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8527.531751] Lustre: lustre-OST0000: new disk, initializing [ 8527.534252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8527.538624] Lustre: 350603:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8527.598616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8531.219230] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8533.017420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8533.024756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8533.088504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8537.265116] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8558.563060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8558.569457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8558.575250] Lustre: Skipped 1 previous similar message [ 8558.585269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8562.535097] Lustre: server umount lustre-OST0000 complete [ 8566.240347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8566.242785] LustreError: Skipped 1 previous similar message [ 8566.245201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8566.250514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8566.252547] Lustre: Skipped 1 previous similar message [ 8568.805543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.325055] Lustre: server umount lustre-MDT0000 complete [ 8573.239501] LustreError: 348633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488985 with bad export cookie 11843931444752262658 [ 8573.245818] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8573.246372] LustreError: 348633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8573.410817] Lustre: server umount lustre-MDT0001 complete [ 8580.208320] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8581.907664] Key type lgssc unregistered [ 8582.115341] LNet: 352753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8582.119922] LNetError: 352753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8582.129914] LNet: Removed LNI 192.168.201.136@tcp [ 8582.700761] Key type .llcrypt unregistered [ 8582.702980] Key type ._llcrypt unregistered [ 8639.540657] Key type ._llcrypt registered [ 8639.541853] Key type .llcrypt registered [ 8639.610809] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8639.799423] Key type .llcrypt unregistered [ 8639.801582] Key type ._llcrypt unregistered [ 8693.412918] Key type ._llcrypt registered [ 8693.414257] Key type .llcrypt registered [ 8693.767886] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8694.656653] Key type .llcrypt unregistered [ 8694.670035] Key type ._llcrypt unregistered [ 8753.326458] Key type ._llcrypt registered [ 8753.330300] Key type .llcrypt registered [ 8753.490230] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8753.887451] Key type .llcrypt unregistered [ 8753.891677] Key type ._llcrypt unregistered [ 8780.233358] Key type ._llcrypt registered [ 8780.237313] Key type .llcrypt registered [ 8780.350683] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8780.603296] Key type .llcrypt unregistered [ 8780.605888] Key type ._llcrypt unregistered [ 8875.606727] Key type ._llcrypt registered [ 8875.611507] Key type .llcrypt registered [ 8875.756682] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8876.108470] Key type .llcrypt unregistered [ 8876.113204] Key type ._llcrypt unregistered [ 8908.380351] Key type ._llcrypt registered [ 8908.386646] Key type .llcrypt registered [ 8908.536279] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8908.787884] Key type .llcrypt unregistered [ 8908.791473] Key type ._llcrypt unregistered [ 8941.039890] Key type ._llcrypt registered [ 8941.042740] Key type .llcrypt registered [ 8941.203545] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8941.662338] Key type .llcrypt unregistered [ 8941.665830] Key type ._llcrypt unregistered [ 8974.329129] Key type ._llcrypt registered [ 8974.332649] Key type .llcrypt registered [ 8974.470924] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8974.707564] Key type .llcrypt unregistered [ 8974.710574] Key type ._llcrypt unregistered [ 9010.545575] Key type ._llcrypt registered [ 9010.552181] Key type .llcrypt registered [ 9010.839461] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9011.549345] Key type .llcrypt unregistered [ 9011.562308] Key type ._llcrypt unregistered [ 9054.579748] Key type ._llcrypt registered [ 9054.582182] Key type .llcrypt registered [ 9054.721813] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9055.095417] Key type .llcrypt unregistered [ 9055.099379] Key type ._llcrypt unregistered [ 9120.206950] Key type ._llcrypt registered [ 9120.209750] Key type .llcrypt registered [ 9120.310587] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9120.569299] Key type .llcrypt unregistered [ 9120.572638] Key type ._llcrypt unregistered [ 9151.519704] Key type ._llcrypt registered [ 9151.521935] Key type .llcrypt registered [ 9151.665650] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9151.922255] Key type .llcrypt unregistered [ 9151.923815] Key type ._llcrypt unregistered [ 9184.191486] Key type ._llcrypt registered [ 9184.193529] Key type .llcrypt registered [ 9184.287232] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9184.597518] Key type .llcrypt unregistered [ 9184.600297] Key type ._llcrypt unregistered [ 9242.703272] Key type ._llcrypt registered [ 9242.708314] Key type .llcrypt registered [ 9242.863291] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9263.559410] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9265.062887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9265.176742] alg: No test for adler32 (adler32-zlib) [ 9266.569915] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 9266.977871] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9268.866361] Key type lgssc registered [ 9270.953397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9305.545733] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9305.585096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9307.103638] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 9307.176846] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 9307.349858] Lustre: tst250fs-MDT0000: new disk, initializing [ 9307.480343] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9307.535199] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9314.913814] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9331.721018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9331.985811] Lustre: 373027:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9332.135809] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 9332.144360] Lustre: Skipped 1 previous similar message [ 9332.342742] Lustre: tst250fs-MDT0001: new disk, initializing [ 9332.490668] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9332.587391] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9332.608550] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9339.067165] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9345.373451] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9353.181078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9353.588920] Lustre: tst250fs-OST0000: new disk, initializing [ 9353.593684] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 9353.604530] Lustre: 374658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9353.737427] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9361.784772] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9363.571274] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9363.579316] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9363.719787] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9373.692490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9373.869568] Lustre: tst250fs-OST0001: new disk, initializing [ 9373.875230] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 9373.883660] Lustre: 375531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9373.996129] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9381.125608] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9381.566816] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9381.581241] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9381.763827] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9386.495952] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 9389.835767] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [ 9389.842961] Lustre: Skipped 1 previous similar message [ 9417.703435] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9417.710555] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9417.735888] Lustre: Skipped 3 previous similar messages [ 9417.758111] Lustre: Skipped 3 previous similar messages [ 9421.333503] Lustre: server umount tst250fs-MDT0000 complete [ 9425.814432] LustreError: 373018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489838 with bad export cookie 15576276914231218500 [ 9425.823410] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9425.841120] LustreError: 373018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9426.626358] Lustre: server umount tst250fs-MDT0001 complete [ 9442.358781] Lustre: server umount tst250fs-OST0000 complete [ 9444.319144] Lustre: 371247:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782489840/real 1782489840] req@ffff9eef511fb100 x1869075897722112/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782489856 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9444.389045] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9447.396612] Lustre: 371249:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782489843/real 1782489843] req@ffff9eef44213800 x1869075897722368/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782489859 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9448.421223] Lustre: 371249:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782489844/real 1782489844] req@ffff9eef44210000 x1869075897722624/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782489860 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9448.921020] Lustre: server umount tst250fs-OST0001 complete [ 9476.161892] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9486.632913] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9530.311991] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:05:40 (1782489940) [ 9531.801466] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 9548.940494] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9563.375397] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9596.502981] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9216 sec ========== 12:06:47 (1782490007) [ 9598.331836] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:06:49 (1782490009) === [ 9601.948621] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:06:52 (1782490012) === [ 9647.383604] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9651.395344] Key type lgssc unregistered [ 9651.674369] LNet: 384682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9651.691271] LNetError: 384682:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9652.711952] LNet: Removed LNI 192.168.201.136@tcp [ 9653.812841] Key type .llcrypt unregistered [ 9653.817289] Key type ._llcrypt unregistered