[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506279108 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002412] x2apic enabled [ 0.003014] Switched APIC routing to physical x2apic. [ 0.004022] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.011049] LSM: Security Framework initializing [ 0.012092] Yama: becoming mindful. [ 0.014061] SELinux: Initializing. [ 0.016033] *** VALIDATE selinux *** [ 0.024713] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029073] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031142] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033155] *** VALIDATE tmpfs *** [ 0.034526] *** VALIDATE proc *** [ 0.035332] *** VALIDATE cgroup *** [ 0.036017] *** VALIDATE cgroup2 *** [ 0.037302] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038184] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040043] Spectre V2 : User space: Vulnerable [ 0.042007] Speculative Store Bypass: Vulnerable [ 0.045830] debug: unmapping init [mem 0xffffffffbce59000-0xffffffffbce60fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048785] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049034] ... version: 2 [ 0.050026] ... bit width: 48 [ 0.051021] ... generic registers: 4 [ 0.052024] ... value mask: 0000ffffffffffff [ 0.053026] ... max period: 00007fffffffffff [ 0.054024] ... fixed-purpose events: 3 [ 0.055020] ... event mask: 000000070000000f [ 0.056382] rcu: Hierarchical SRCU implementation. [ 0.058910] smp: Bringing up secondary CPUs ... [ 0.059739] x86: Booting SMP configuration: [ 0.060046] .... node #0, CPUs: #1 #2 #3 [ 0.064370] smp: Brought up 1 node, 4 CPUs [ 0.066028] smpboot: Max logical packages: 1 [ 0.067023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141020] node 0 deferred pages initialised in 70ms [ 0.145187] devtmpfs: initialized [ 0.146270] x86/mm: Memory block size: 128MB [ 0.148928] gcov: version magic: 0x41383552 [ 0.150240] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155421] pinctrl core: initialized pinctrl subsystem [ 0.158321] [ 0.159009] ************************************************************* [ 0.163015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166015] ** ** [ 0.168018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171014] ** ** [ 0.172000] ** This means that this kernel is built to expose internal ** [ 0.174016] ** IOMMU data structures, which may compromise security on ** [ 0.176013] ** your system. ** [ 0.179016] ** ** [ 0.181014] ** If you see this message and you are not debugging the ** [ 0.184015] ** kernel, report this immediately to your vendor! ** [ 0.186013] ** ** [ 0.189083] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191014] ************************************************************* [ 0.195070] NET: Registered protocol family 16 [ 0.197537] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205023] cpuidle: using governor menu [ 0.206660] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208489] PCI: Using configuration type 1 for base access [ 0.210124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218177] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221100] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224100] cryptd: max_cpu_qlen set to 1000 [ 0.227311] ACPI: Added _OSI(Module Device) [ 0.228022] ACPI: Added _OSI(Processor Device) [ 0.230018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237151] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242484] ACPI: Interpreter enabled [ 0.244051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245011] ACPI: Using IOAPIC for interrupt routing [ 0.246111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249452] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.259000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.265025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.275481] acpiphp: Slot [2] registered [ 0.277146] acpiphp: Slot [5] registered [ 0.281245] acpiphp: Slot [6] registered [ 0.284145] acpiphp: Slot [7] registered [ 0.285205] acpiphp: Slot [8] registered [ 0.287158] acpiphp: Slot [9] registered [ 0.289199] acpiphp: Slot [10] registered [ 0.290315] acpiphp: Slot [3] registered [ 0.292192] acpiphp: Slot [4] registered [ 0.294125] acpiphp: Slot [11] registered [ 0.296144] acpiphp: Slot [12] registered [ 0.297129] acpiphp: Slot [13] registered [ 0.299142] acpiphp: Slot [14] registered [ 0.301188] acpiphp: Slot [15] registered [ 0.302156] acpiphp: Slot [16] registered [ 0.304150] acpiphp: Slot [17] registered [ 0.306151] acpiphp: Slot [18] registered [ 0.307216] acpiphp: Slot [19] registered [ 0.309133] acpiphp: Slot [20] registered [ 0.311185] acpiphp: Slot [21] registered [ 0.312201] acpiphp: Slot [22] registered [ 0.314153] acpiphp: Slot [23] registered [ 0.316173] acpiphp: Slot [24] registered [ 0.317128] acpiphp: Slot [25] registered [ 0.319129] acpiphp: Slot [26] registered [ 0.320133] acpiphp: Slot [27] registered [ 0.322177] acpiphp: Slot [28] registered [ 0.324147] acpiphp: Slot [29] registered [ 0.325146] acpiphp: Slot [30] registered [ 0.327152] acpiphp: Slot [31] registered [ 0.329130] PCI host bridge to bus 0000:00 [ 0.331031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342051] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349108] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.358020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.363066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372041] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376267] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379866] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382066] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385914] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.408049] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.416019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.420952] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.431022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.440025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.454023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.465199] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.480029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.514835] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.522029] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.539024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.561025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.574175] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.586025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.596034] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.618028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.630523] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.638022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.648022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.669021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.680105] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.687020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.692019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.708032] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.718166] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.723411] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.725381] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.727282] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.729136] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.732157] iommu: Default domain type: Passthrough [ 0.735578] SCSI subsystem initialized [ 0.737169] ACPI: bus type USB registered [ 0.739180] usbcore: registered new interface driver usbfs [ 0.741120] usbcore: registered new interface driver hub [ 0.743119] usbcore: registered new device driver usb [ 0.745250] pps_core: LinuxPPS API ver. 1 registered [ 0.748018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.751090] PTP clock support registered [ 0.754135] EDAC MC: Ver: 3.0.0 [ 0.755678] PCI: Using ACPI for IRQ routing [ 0.757040] NetLabel: Initializing [ 0.758014] NetLabel: domain hash size = 128 [ 0.759014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.760193] NetLabel: unlabeled traffic allowed by default [ 0.763054] vgaarb: loaded [ 0.764317] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.766024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.773219] clocksource: Switched to clocksource kvm-clock [ 0.883576] VFS: Disk quotas dquot_6.6.0 [ 0.885236] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.888374] *** VALIDATE ramfs *** [ 0.889678] *** VALIDATE hugetlbfs *** [ 0.892270] pnp: PnP ACPI init [ 0.894885] pnp: PnP ACPI: found 6 devices [ 0.913991] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.917587] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.920242] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.922350] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.925297] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.928113] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.931268] NET: Registered protocol family 2 [ 0.934161] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.938511] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.942455] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.947439] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.953076] TCP: Hash tables configured (established 65536 bind 65536) [ 0.956663] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.960221] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.963446] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.966800] NET: Registered protocol family 1 [ 0.971466] RPC: Registered named UNIX socket transport module. [ 0.974094] RPC: Registered udp transport module. [ 0.976123] RPC: Registered tcp transport module. [ 0.978263] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.981220] NET: Registered protocol family 44 [ 0.983321] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.985830] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.988081] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.990744] PCI: CLS 0 bytes, default 64 [ 0.993183] Unpacking initramfs... [ 2.436321] debug: unmapping init [mem 0xffff8d1f7cc54000-0xffff8d1f7ffbffff] [ 2.440515] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.443313] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.446698] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.948771] Initialise system trusted keyrings [ 2.950773] Key type blacklist registered [ 2.952964] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.963231] zbud: loaded [ 2.966869] *** VALIDATE nfs *** [ 2.968233] *** VALIDATE nfs4 *** [ 2.970147] pstore: using deflate compression [ 2.973523] Platform Keyring initialized [ 3.069780] NET: Registered protocol family 38 [ 3.071392] Key type asymmetric registered [ 3.072757] Asymmetric key parser 'x509' registered [ 3.074718] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.077801] io scheduler mq-deadline registered [ 3.079304] io scheduler kyber registered [ 3.080986] io scheduler bfq registered [ 3.083625] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.086926] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.090110] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.093380] ACPI: Power Button [PWRF] [ 3.098903] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.105432] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.121902] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.130971] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.148946] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.178822] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.207161] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.212064] Non-volatile memory driver v1.3 [ 3.214038] Linux agpgart interface v0.103 [ 3.246801] virtio_blk virtio1: [vda] 145208 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.250204] vda: detected capacity change from 0 to 74346496 [ 3.267668] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.270238] vdb: detected capacity change from 0 to 1073741824 [ 3.291659] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.295940] vdc: detected capacity change from 0 to 2621440000 [ 3.316128] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.319244] vdd: detected capacity change from 0 to 2621440000 [ 3.337303] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.340965] vde: detected capacity change from 0 to 4294967296 [ 3.359311] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.363276] vdf: detected capacity change from 0 to 4294967296 [ 3.372226] libphy: Fixed MDIO Bus: probed [ 3.382451] usbcore: registered new interface driver usbserial_generic [ 3.385259] usbserial: USB Serial support registered for generic [ 3.387829] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.392480] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.394184] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.396646] mousedev: PS/2 mouse device common for all mice [ 3.400064] rtc_cmos 00:05: RTC can wake from S4 [ 3.402907] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.403746] rtc_cmos 00:05: registered as rtc0 [ 3.410863] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.412375] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.420230] intel_pstate: CPU model not supported [ 3.420836] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.426984] hid: raw HID events driver (C) Jiri Kosina [ 3.429354] usbcore: registered new interface driver usbhid [ 3.431541] usbhid: USB HID core driver [ 3.433326] drop_monitor: Initializing network drop monitor service [ 3.437422] Initializing XFRM netlink socket [ 3.440362] NET: Registered protocol family 10 [ 3.443860] Segment Routing with IPv6 [ 3.446180] NET: Registered protocol family 17 [ 3.448924] mpls_gso: MPLS GSO support [ 3.455343] RAS: Correctable Errors collector initialized. [ 3.458062] AVX version of gcm_enc/dec engaged. [ 3.459612] AES CTR mode by8 optimization enabled [ 3.541066] sched_clock: Marking stable (3541035484, 0)->(4596829140, -1055793656) [ 3.544681] registered taskstats version 1 [ 3.546753] Loading compiled-in X.509 certificates [ 3.548818] zswap: loaded using pool lzo/zbud [ 3.575680] Key type big_key registered [ 3.588525] Key type encrypted registered [ 3.589895] ima: No TPM chip found, activating TPM-bypass! [ 3.591496] ima: Allocated hash algorithm: sha1 [ 3.592866] ima: No architecture policies found [ 3.594189] evm: Initialising EVM extended attributes: [ 3.595745] evm: security.selinux [ 3.596732] evm: security.ima [ 3.597617] evm: security.capability [ 3.598627] evm: HMAC attrs: 0x1 [ 3.600620] rtc_cmos 00:05: setting system clock to 2026-07-04 03:43:39 UTC (1783136619) [ 3.606421] debug: unmapping init [mem 0xffffffffbde03000-0xffffffffbdffffff] [ 3.608890] debug: unmapping init [mem 0xffffffffbcb82000-0xffffffffbce58fff] [ 3.615129] Write protecting the kernel read-only data: 28672k [ 3.618028] debug: unmapping init [mem 0xffffffffbb203000-0xffffffffbb3fffff] [ 3.620350] debug: unmapping init [mem 0xffffffffbbb14000-0xffffffffbbbfffff] [ 3.688524] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.696962] systemd[1]: Detected virtualization kvm. [ 3.698970] systemd[1]: Detected architecture x86-64. [ 3.700746] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.725652] systemd[1]: No hostname configured. [ 3.727143] systemd[1]: Set hostname to . [ 3.729042] random: systemd: uninitialized urandom read (16 bytes read) [ 3.731147] systemd[1]: Initializing machine ID from random generator. [ 3.785407] random: ln: uninitialized urandom read (6 bytes read) [ 3.872261] random: systemd: uninitialized urandom read (16 bytes read) [ 3.876705] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.886036] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 3.895233] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.513808] device-mapper: uevent: version 1.0.3 [ 4.516798] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.264155] virtio_net virtio0 ens2: renamed from eth0 [ 5.332464] scsi host0: ata_piix [ 5.375694] scsi host1: ata_piix [ 5.393440] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.398804] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 6.697881] random: fast init done [ 9.213849] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.103505] random: crng init done [ 10.105201] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.661190] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.895112] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.189730] SELinux: Disabled at runtime. [ 12.254940] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.266037] systemd[1]: Detected virtualization kvm. [ 12.268279] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.856086] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.860445] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.866904] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.871898] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.876952] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.885533] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.894363] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ 12.986555] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.374554] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.652334] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.757748] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.911251] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.928303] EDAC sbridge: Ver: 1.1.2 [ 15.639542] Key type dns_resolver registered [ 15.949209] NFS: Registering the id_resolver key type [ 15.951533] Key type id_resolver registered [ 15.953368] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg448-server login: [ 24.338895] hrtimer: interrupt took 5957244 ns [ 67.454811] libcfs: loading out-of-tree module taints kernel. [ 67.520817] Key type ._llcrypt registered [ 67.525216] Key type .llcrypt registered [ 67.631046] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 84.317460] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 85.701127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 85.729069] alg: No test for adler32 (adler32-zlib) [ 87.070192] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 87.954435] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 89.791260] Key type lgssc registered [ 91.405211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.438173] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.336425] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 154.147462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.188969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.437559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.483797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.602641] Lustre: lustre-MDT0000: new disk, initializing [ 155.694354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.712348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.715420] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.788323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.921046] Lustre: 6506:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.971359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.974534] Lustre: Skipped 1 previous similar message [ 173.082974] Lustre: lustre-MDT0001: new disk, initializing [ 173.183580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.226839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.240717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.572432] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.241943] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 190.357908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.746671] Lustre: lustre-OST0000: new disk, initializing [ 190.752925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.759945] Lustre: 8440:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.866133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.138860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 196.147282] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.262053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.063752] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.819877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.019795] Lustre: lustre-OST0001: new disk, initializing [ 211.031416] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 211.038811] Lustre: 9514:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.138256] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.434658] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.629871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.643925] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.749903] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 228.291916] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 240.397314] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 246.821868] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.085392] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing yml_node [ 255.521322] Lustre: DEBUG MARKER: Client: 2.17.54.85 [ 257.672530] Lustre: DEBUG MARKER: MDS: 2.17.54.85 [ 259.927438] Lustre: DEBUG MARKER: OSS: 2.17.54.85 [ 261.700550] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 3 23:47:55 EDT 2026 [ 277.884252] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 279.310077] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 296.416906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 296.423721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.439622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 299.494223] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.494777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 299.510637] Lustre: Skipped 2 previous similar messages [ 299.527852] Lustre: Skipped 3 previous similar messages [ 301.317992] Lustre: server umount lustre-MDT0000 complete [ 304.889674] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136920 with bad export cookie 14687928294710174585 [ 304.895800] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 304.907686] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 309.733921] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.738103] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.738993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.752621] Lustre: Skipped 1 previous similar message [ 309.780759] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 311.209580] Lustre: server umount lustre-MDT0001 complete [ 315.253366] Lustre: server umount lustre-OST0000 complete [ 319.184565] Lustre: server umount lustre-OST0001 complete [ 325.150516] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 334.413220] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 373.806433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.124948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 374.157887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 374.224231] Lustre: lustre-MDT0000: new disk, initializing [ 374.302928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 374.317143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 378.386489] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 389.544536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.709898] Lustre: 15066:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 389.768644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 389.771922] Lustre: Skipped 1 previous similar message [ 389.857768] Lustre: lustre-MDT0001: new disk, initializing [ 389.960776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 389.995416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 390.005471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 394.875487] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 404.808325] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 410.678349] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 418.636519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.100961] Lustre: lustre-OST0000: new disk, initializing [ 419.108762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 419.121936] Lustre: 17034:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 419.288707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.729232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 420.739465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 420.915823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 427.132893] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 437.289618] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 442.419884] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 442.681343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 447.384969] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 447.648984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 451.553497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.557107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.559059] LustreError: Skipped 1 previous similar message [ 451.575373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.582370] Lustre: Skipped 1 previous similar message [ 451.605242] Lustre: Skipped 2 previous similar messages [ 455.757455] Lustre: server umount lustre-OST0000 complete [ 461.794587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 461.799274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.800484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 461.824960] Lustre: Skipped 1 previous similar message [ 465.953220] Lustre: server umount lustre-MDT0000 complete [ 466.918768] LustreError: 15077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.950187] LustreError: 15077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 470.362102] LustreError: 15057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137086 with bad export cookie 14687928294710176118 [ 470.368671] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.376819] LustreError: 15057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 470.724267] Lustre: server umount lustre-MDT0001 complete [ 477.957554] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:51:31 (1783137091) [ 499.734385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.676198] LustreError: 19910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.811399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.824682] LustreError: 19911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.999217] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 510.947995] LustreError: 19910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.857317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.064748] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 516.078163] Lustre: Skipped 1 previous similar message [ 521.067523] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 529.503955] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 537.411928] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 539.740229] Lustre: Failing over lustre-MDT0000 [ 540.029642] Lustre: server umount lustre-MDT0000 complete [ 541.681493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 541.696424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.714246] LustreError: 19911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.559199] Lustre: 3644:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783137158/real 1783137158] req@ffff8d1ffce52d80 x1869754351644032/t0(0) o400->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783137174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 558.570097] LustreError: 19906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.607525] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.656216] LustreError: 19906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 560.526026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.807134] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcbd600f6cc9beefc [ 568.823116] Lustre: MGC192.168.204.148@tcp: Connection restored to 0@lo (at 0@lo) [ 569.070811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 569.076228] Lustre: Skipped 1 previous similar message [ 573.621724] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 574.448058] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 574.473210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 574.499936] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 584.051219] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 588.780377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.787476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.806503] Lustre: Skipped 2 previous similar messages [ 588.826123] Lustre: Skipped 1 previous similar message [ 594.756414] Lustre: server umount lustre-MDT0000 complete [ 594.914090] LustreError: 19905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.934179] LustreError: 19905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 598.422834] LustreError: 19892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137214 with bad export cookie 14687928294710177532 [ 598.427858] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.436143] LustreError: 19892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 598.855351] Lustre: server umount lustre-MDT0001 complete [ 608.712425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.085876] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 622.333235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.525416] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 635.922680] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 642.467992] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 645.379918] Lustre: Failing over lustre-MDT0000 [ 645.662394] Lustre: server umount lustre-MDT0000 complete [ 647.472512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.497946] Lustre: Skipped 1 previous similar message [ 663.013983] LustreError: 23976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 663.037641] LustreError: 23976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 664.035431] Lustre: 3642:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783137263/real 1783137263] req@ffff8d1ffce52300 x1869754351674496/t0(0) o400->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783137279 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 664.063228] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 664.885290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.330417] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcbd600f6cc9bf2a6 [ 673.348530] Lustre: MGC192.168.204.148@tcp: Connection restored to 0@lo (at 0@lo) [ 673.363237] Lustre: Skipped 1 previous similar message [ 673.661754] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 678.284652] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 678.882199] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 678.897626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 678.918652] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 688.263870] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 694.240352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 694.251687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 694.259541] Lustre: Skipped 2 previous similar messages [ 699.519362] Lustre: server umount lustre-MDT0000 complete [ 704.145757] LustreError: 23230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137320 with bad export cookie 14687928294710178470 [ 704.174664] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.663378] Lustre: server umount lustre-MDT0001 complete [ 716.235155] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:55:29 (1783137329) [ 733.108431] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 743.469358] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 787.064278] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 799.953449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.175992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 800.207538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 800.302308] Lustre: lustre-MDT0000: new disk, initializing [ 800.415532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.423099] Lustre: Skipped 3 previous similar messages [ 800.461936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 805.443715] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 818.753921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.908449] Lustre: 29645:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 819.275389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 825.782800] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 835.716219] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 840.988926] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 848.787344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 849.207391] Lustre: 31614:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 851.146561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 851.205164] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 851.278302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 851.297128] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 856.291423] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 865.159621] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 870.989463] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 897.505610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 897.515656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.522301] LustreError: Skipped 1 previous similar message [ 897.537818] Lustre: Skipped 2 previous similar messages [ 897.555475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 897.569062] Lustre: Skipped 3 previous similar messages [ 900.018072] Lustre: server umount lustre-OST0000 complete [ 910.199381] Lustre: server umount lustre-MDT0000 complete [ 912.865437] LustreError: 29656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 912.890435] LustreError: 29656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 913.971341] LustreError: 30439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137529 with bad export cookie 14687928294710178876 [ 913.974736] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 913.980661] LustreError: 30439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 914.383943] Lustre: server umount lustre-MDT0001 complete [ 924.426875] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 927.731684] Key type lgssc unregistered [ 928.144746] LNet: 33594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.153976] LNetError: 33594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.175888] LNet: Removed LNI 192.168.204.148@tcp [ 929.252549] Key type .llcrypt unregistered [ 929.259487] Key type ._llcrypt unregistered [ 956.231524] Key type ._llcrypt registered [ 956.236946] Key type .llcrypt registered [ 956.378725] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 957.518336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 957.630386] alg: No test for adler32 (adler32-zlib) [ 958.811251] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 959.088898] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 960.855821] Key type lgssc registered [ 962.150864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 972.084081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 972.113102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.833497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.171936] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 984.033833] LustreError: 35009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.056559] LustreError: 35009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 989.157461] LustreError: 35008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.929164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 991.525398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 995.989422] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1004.467690] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1011.722658] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1019.257906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1019.735441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.337782] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1034.770955] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1037.138277] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1043.296063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1061.846739] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:01:15 (1783137675) [ 1063.782808] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1066.142267] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1074.277414] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:01:27 (1783137687) [ 1085.184289] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:01:38 (1783137698) [ 1086.400882] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1096.574456] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1096.581326] Lustre: Skipped 2 previous similar messages [ 1098.838779] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:01:52 (1783137712) [ 1100.964837] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1100.971689] Lustre: Skipped 2 previous similar messages [ 1110.341654] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1110.352652] Lustre: Skipped 4 previous similar messages [ 1125.395348] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:02:18 (1783137738) [ 1128.619337] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1132.152814] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1141.856505] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:02:35 (1783137755) [ 1144.063930] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1144.085937] Lustre: Skipped 6 previous similar messages [ 1158.986867] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:02:52 (1783137772) [ 1175.520775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1175.534687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.551178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1176.547552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1176.554444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1176.560722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.583598] Lustre: Skipped 1 previous similar message [ 1181.671689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.682135] Lustre: Skipped 1 previous similar message [ 1186.796768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1186.802991] Lustre: Skipped 1 previous similar message [ 1189.343220] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1189.440571] Lustre: server umount lustre-OST0000 complete [ 1196.519537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1196.532264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.540698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.957709] Lustre: server umount lustre-MDT0000 complete [ 1202.145448] LustreError: 37686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.182769] LustreError: 37686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1205.179509] LustreError: 38608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137821 with bad export cookie 6443643265605489989 [ 1205.195861] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1205.203544] LustreError: 38608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1205.862812] Lustre: server umount lustre-MDT0001 complete [ 1220.552432] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1224.639870] Key type lgssc unregistered [ 1224.965483] LNet: 42905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.983257] LNetError: 42905:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.015861] LNet: Removed LNI 192.168.204.148@tcp [ 1226.327755] Key type .llcrypt unregistered [ 1226.336973] Key type ._llcrypt unregistered [ 1246.238567] Key type ._llcrypt registered [ 1246.244969] Key type .llcrypt registered [ 1246.391666] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 1264.859338] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1266.846229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1266.897938] alg: No test for adler32 (adler32-zlib) [ 1268.146848] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 1268.665465] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1270.375479] Key type lgssc registered [ 1272.226842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1315.242090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1315.280717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1316.601421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1316.656917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1316.764880] Lustre: lustre-MDT0000: new disk, initializing [ 1316.862717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.875666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1321.851307] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1334.065411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.261221] Lustre: 46069:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1334.376849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1334.385832] Lustre: Skipped 1 previous similar message [ 1334.542322] Lustre: lustre-MDT0001: new disk, initializing [ 1334.681243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1334.731922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1334.746776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1340.080795] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1351.393782] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1359.026118] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1371.004048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1371.490881] Lustre: lustre-OST0000: new disk, initializing [ 1371.496561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1371.507318] Lustre: 48039:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1371.667948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.483125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1380.485566] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1380.499049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1380.660740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1392.123963] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1398.392984] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1398.655878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1403.759125] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1404.094732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1411.558471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.562963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1411.564918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1411.583772] Lustre: Skipped 1 previous similar message [ 1411.619711] LustreError: Skipped 1 previous similar message [ 1413.048337] Lustre: server umount lustre-OST0000 complete [ 1421.793633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.794290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1421.830712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1421.833669] Lustre: Skipped 1 previous similar message [ 1423.623491] Lustre: server umount lustre-MDT0000 complete [ 1429.041312] LustreError: 46060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138044 with bad export cookie 5162550237716249507 [ 1429.043839] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1429.049233] LustreError: 46060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1429.773656] Lustre: server umount lustre-MDT0001 complete [ 1443.410109] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:07:37 (1783138057) [ 1452.118596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.876915] LustreError: 50499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.909479] LustreError: 50499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1452.985272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.159025] LustreError: 50500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1458.233738] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1462.241652] LustreError: 50499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.365552] LustreError: 50500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1468.928899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.440219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1473.930878] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1483.519544] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1493.787835] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1503.168777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1503.638670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.948667] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1518.381318] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1527.961362] LustreError: 53028:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1529.093289] Lustre: Modifying parameter general.timeout=1 in log params [ 1530.227288] Lustre: Modifying parameter general.timeout=2 in log params [ 1533.013491] Lustre: Modifying parameter general.timeout=5 in log params [ 1533.017530] Lustre: Skipped 2 previous similar messages [ 1537.547477] Lustre: Modifying parameter general.timeout=10 in log params [ 1537.553499] Lustre: Skipped 4 previous similar messages [ 1545.853348] Lustre: Modifying parameter general.timeout=19 in log params [ 1545.857566] Lustre: Skipped 8 previous similar messages [ 1562.295445] Lustre: Modifying parameter general.timeout=36 in log params [ 1562.298934] Lustre: Skipped 16 previous similar messages [ 1595.155520] Lustre: Modifying parameter general.timeout=65 in log params [ 1595.158413] Lustre: Skipped 28 previous similar messages [ 1659.709099] Lustre: Modifying parameter general.timeout=124 in log params [ 1659.717175] Lustre: Skipped 58 previous similar messages [ 1678.203958] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:11:32 (1783138292) [ 1681.566857] Lustre: 59986:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1683.801259] Lustre: 60095:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1686.040355] Lustre: 60193:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1690.520356] Lustre: 60390:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1690.532459] Lustre: 60390:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 1725.741982] Lustre: 61867:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1725.754216] Lustre: 61867:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 1725.781337] LustreError: 61867:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1731.288128] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:12:25 (1783138345) [ 1751.012933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1751.039555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.050767] Lustre: Skipped 1 previous similar message [ 1751.061914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1755.496923] Lustre: server umount lustre-MDT0000 complete [ 1756.649870] LustreError: 50499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1756.668817] LustreError: 50499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1760.388314] LustreError: 53077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138376 with bad export cookie 5162550237716250389 [ 1760.401731] LustreError: 53077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1760.403036] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1761.039890] Lustre: server umount lustre-MDT0001 complete [ 1777.835836] Lustre: server umount lustre-OST0000 complete [ 1813.534359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.447207] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1814.556840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1814.599809] Lustre: Skipped 28 previous similar messages [ 1815.093705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.947368] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1836.556603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.681918] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1836.717717] Lustre: 64402:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1843.297322] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1853.577474] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1862.243122] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1870.557521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1870.968677] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1871.333041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.336400] Lustre: Skipped 1 previous similar message [ 1880.164789] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1891.335699] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1894.182536] Lustre: 66974:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1905.648808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1905.657676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.662248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1905.670981] Lustre: Skipped 3 previous similar messages [ 1905.675351] Lustre: Skipped 2 previous similar messages [ 1909.944873] Lustre: server umount lustre-OST0000 complete [ 1915.880124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.893093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.899614] Lustre: Skipped 1 previous similar message [ 1915.910929] Lustre: Skipped 2 previous similar messages [ 1920.993378] LustreError: 64408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.035936] LustreError: 64408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1921.051355] Lustre: server umount lustre-MDT0000 complete [ 1926.338186] LustreError: 64393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138542 with bad export cookie 5162550237716253637 [ 1926.347253] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1926.361979] LustreError: 64393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1927.073455] Lustre: server umount lustre-MDT0001 complete [ 1940.474372] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1944.755793] Key type lgssc unregistered [ 1945.232735] LNet: 68183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.240023] LNetError: 68183:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.274684] LNet: Removed LNI 192.168.204.148@tcp [ 1946.486360] Key type .llcrypt unregistered [ 1946.489659] Key type ._llcrypt unregistered [ 1974.270086] Key type ._llcrypt registered [ 1974.280132] Key type .llcrypt registered [ 1974.445922] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1975.845403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1975.855319] alg: No test for adler32 (adler32-zlib) [ 1977.045321] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 1977.426742] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1979.247276] Key type lgssc registered [ 1980.790683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1989.925466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1989.962024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.584402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.002301] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2001.891823] LustreError: 69598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.920085] LustreError: 69598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2007.013087] LustreError: 69599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.139847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.801308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2014.182648] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2023.813122] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2029.995100] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2039.388293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2040.211356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.017688] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2056.895988] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2070.849807] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2085.348446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2085.356250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.365237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2087.406657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.407834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2087.423856] Lustre: Skipped 1 previous similar message [ 2087.457458] Lustre: Skipped 2 previous similar messages [ 2088.722275] Lustre: server umount lustre-MDT0000 complete [ 2092.514015] LustreError: 71194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2092.544470] LustreError: 71194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2094.159375] LustreError: 69579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138710 with bad export cookie 12001617826037072237 [ 2094.180342] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2094.681734] Lustre: server umount lustre-MDT0001 complete [ 2109.340523] Lustre: server umount lustre-OST0000 complete [ 2139.704782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2140.068251] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2140.120215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2140.573644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.757865] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2155.653146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2155.822517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2155.890309] Lustre: 74471:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2156.482849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2161.266222] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2170.375050] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2177.054056] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2184.940488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2185.299028] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2185.694469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.279515] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2201.412771] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2203.227763] Lustre: 77048:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2205.546365] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2215.393535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.398209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2215.398369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2215.409736] Lustre: Skipped 1 previous similar message [ 2215.447425] LustreError: Skipped 1 previous similar message [ 2218.250476] Lustre: server umount lustre-OST0000 complete [ 2223.074393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2223.084968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.118632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.121674] Lustre: Skipped 1 previous similar message [ 2228.772630] Lustre: server umount lustre-MDT0000 complete [ 2230.764651] LustreError: 74482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.777807] LustreError: 74482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2233.186828] LustreError: 74462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138849 with bad export cookie 12001617826037073084 [ 2233.187886] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2233.194101] LustreError: 74462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2233.582897] Lustre: server umount lustre-MDT0001 complete [ 2246.183337] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2249.592255] Key type lgssc unregistered [ 2249.909897] LNet: 78265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2249.926560] LNetError: 78265:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.947515] LNet: Removed LNI 192.168.204.148@tcp [ 2251.272497] Key type .llcrypt unregistered [ 2251.280759] Key type ._llcrypt unregistered [ 2280.504979] Key type ._llcrypt registered [ 2280.508602] Key type .llcrypt registered [ 2280.687508] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2281.861344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2281.946364] alg: No test for adler32 (adler32-zlib) [ 2283.101977] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 2283.364329] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2285.064292] Key type lgssc registered [ 2286.855670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2297.100579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2297.126018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.916878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.964797] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2309.093610] LustreError: 79681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.125695] LustreError: 79681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2313.203784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.098181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2318.892430] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2327.065794] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2333.347727] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2340.681451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2341.163315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.046125] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2354.079270] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2364.062848] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2364.909414] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2366.625933] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2366.633551] Lustre: Skipped 1 previous similar message [ 2369.474451] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2369.484288] Lustre: Skipped 2 previous similar messages [ 2374.142722] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2374.149142] Lustre: Skipped 4 previous similar messages [ 2382.572698] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2382.577926] Lustre: Skipped 8 previous similar messages [ 2398.802883] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2398.811653] Lustre: Skipped 17 previous similar messages [ 2431.875502] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2431.880746] Lustre: Skipped 34 previous similar messages [ 2467.538164] Lustre: 87781:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2470.620849] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:24:44 (1783139084) [ 2472.588408] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2474.900425] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:24:48 (1783139088) [ 2481.641296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.654779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2481.670224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2481.685121] Lustre: Skipped 1 previous similar message [ 2485.695404] Lustre: server umount lustre-OST0000 complete [ 2491.879972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.897178] Lustre: Skipped 2 previous similar messages [ 2491.898819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2491.912489] Lustre: Skipped 1 previous similar message [ 2495.901694] Lustre: server umount lustre-MDT0000 complete [ 2496.992910] LustreError: 80412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.017043] LustreError: 80412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2500.297279] LustreError: 79660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139116 with bad export cookie 4249512960018455369 [ 2500.302043] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2500.315367] LustreError: 79660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2500.682610] Lustre: server umount lustre-MDT0001 complete [ 2513.796669] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2517.816662] Key type lgssc unregistered [ 2518.311926] LNet: 89040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2518.319672] LNetError: 89040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2518.348066] LNet: Removed LNI 192.168.204.148@tcp [ 2519.885769] Key type .llcrypt unregistered [ 2519.891691] Key type ._llcrypt unregistered [ 2526.364708] Key type ._llcrypt registered [ 2526.368941] Key type .llcrypt registered [ 2526.533946] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2532.114775] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules [ 2533.434749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2533.651758] alg: No test for adler32 (adler32-zlib) [ 2534.877925] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 2535.057929] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2536.791269] Key type lgssc registered [ 2538.260817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2539.549681] LustreError: 90257:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2560.776935] LustreError: 90257:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2561.671846] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2571.964968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2571.988927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.760215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.942019] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2584.032970] LustreError: 91267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.052744] LustreError: 91267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2589.158756] LustreError: 91266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.255420] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:26:42 (1783139202) [ 2594.282045] LustreError: 91267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.872078] LustreError: 91298:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2594.888439] LustreError: 91298:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2594.896833] LustreError: 91298:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2595.414660] Lustre: server umount lustre-MDT0000 complete [ 2609.484966] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2612.235982] Key type lgssc unregistered [ 2612.523783] LNet: 92791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2612.539372] LNetError: 92791:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2612.567341] LNet: Removed LNI 192.168.204.148@tcp [ 2613.691243] Key type .llcrypt unregistered [ 2613.694546] Key type ._llcrypt unregistered [ 2630.744919] Key type ._llcrypt registered [ 2630.747602] Key type .llcrypt registered [ 2630.864390] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2631.903776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2631.942653] alg: No test for adler32 (adler32-zlib) [ 2633.014591] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 2633.311814] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2634.999246] Key type lgssc registered [ 2636.077531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2642.511175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2642.535542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.048711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2648.271717] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2653.158060] LustreError: 94010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.172144] LustreError: 94010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2655.924585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.481418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2660.396406] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2667.924482] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2673.307959] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2681.072548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2681.820929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.353409] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2697.082750] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2748.027439] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:29:21 (1783139361) [ 2750.126949] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2751.992393] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:29:25 (1783139365) [ 2758.626179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2758.637523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.651185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.674508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.681404] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.273589] Lustre: server umount lustre-MDT0000 complete [ 2765.801386] LustreError: 94010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.837284] LustreError: 94010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2765.920209] LustreError: 93991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139381 with bad export cookie 7609082143625420256 [ 2765.935142] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2765.937600] LustreError: 93991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2766.370548] Lustre: server umount lustre-MDT0001 complete [ 2780.872725] Lustre: server umount lustre-OST0000 complete [ 2792.373312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.838504] LustreError: 98306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.013169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.538830] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2798.052043] LustreError: 98307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.155368] LustreError: 98306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2806.053022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.778264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2812.669413] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2820.310963] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2827.168523] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2842.002122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2842.250358] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2842.255158] LustreError: 98296:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2842.281539] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2842.287969] LustreError: 100390:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2842.293789] LustreError: 100390:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2842.298496] LustreError: 100390:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2842.370662] Lustre: server umount lustre-OST0000 complete [ 2842.378784] LustreError: 100390:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2851.437599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.685057] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2851.698726] LustreError: 98296:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2851.742275] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2851.759677] LustreError: 100856:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2851.768459] LustreError: 100861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.773081] LustreError: 100856:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2851.773101] LustreError: 100856:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2851.846432] Lustre: server umount lustre-OST0000 complete [ 2851.849678] LustreError: 100856:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2862.406769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2862.630584] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2862.636572] Lustre: Found index 0 for lustre-OST0000, updating log [ 2862.645795] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2862.729243] Lustre: lustre-OST0000: new disk, initializing [ 2862.734940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2862.743593] Lustre: 101390:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2862.927933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2869.378208] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2873.014729] Lustre: Failing over lustre-OST0000 [ 2873.292376] Lustre: server umount lustre-OST0000 complete [ 2877.922313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2877.928641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.945219] Lustre: Skipped 1 previous similar message [ 2877.949698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2877.954077] Lustre: Skipped 2 previous similar messages [ 2881.508242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2881.516583] Lustre: Skipped 3 previous similar messages [ 2886.625025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2886.634793] Lustre: Skipped 1 previous similar message [ 2891.231137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2891.488978] Lustre: server umount lustre-MDT0000 complete [ 2891.754197] LustreError: 98301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.771450] LustreError: 98301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2895.785590] LustreError: 98288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139511 with bad export cookie 7609082143625423000 [ 2895.786799] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2895.793417] LustreError: 98288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2896.204407] Lustre: server umount lustre-MDT0001 complete [ 2906.034991] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:31:59 (1783139519) [ 2923.370467] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2935.338530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2935.700078] LustreError: 103738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.800768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.796219] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2949.463354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.845954] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2957.892111] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2966.194909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2967.110114] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2974.096318] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2974.261438] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2974.285903] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2974.316389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2979.302058] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2979.329622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2980.332889] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2980.374869] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2980.376852] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2980.380712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2983.103797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2983.196910] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2983.266957] Lustre: lustre-OST0001: new disk, initializing [ 2983.270904] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2983.277502] Lustre: 106057:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2983.504017] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2983.514025] Lustre: Skipped 2 previous similar messages [ 2989.740230] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2993.273686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2993.284264] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2993.388597] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2997.664859] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3003.403651] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3009.003399] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.005871] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3009.036299] Lustre: Skipped 3 previous similar messages [ 3009.050768] Lustre: Skipped 2 previous similar messages [ 3022.304768] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3022.580622] Lustre: server umount lustre-MDT0001 complete [ 3023.243407] LustreError: 103733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.268059] LustreError: 103733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3034.035566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.153845] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3034.159015] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3034.231483] Lustre: 103728:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3035.965779] LustreError: 107614:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3035.977919] LustreError: 107614:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3035.986846] LustreError: 103730:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3036.089802] LustreError: 107629:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.148@tcp: cfg command failed: rc = -17 [ 3036.098520] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3036.098520] [ 3039.964803] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3040.253193] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3050.932242] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:34:24 (1783139664) [ 3069.929796] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3087.761192] Lustre: 109675:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3109.508594] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3113.503944] Lustre: 110808:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3203.555246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3203.568909] LustreError: Skipped 1 previous similar message [ 3203.574768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.598411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3203.601236] Lustre: Skipped 8 previous similar messages [ 3208.566434] Lustre: server umount lustre-MDT0000 complete [ 3209.186501] LustreError: 103735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.205168] LustreError: 103735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3212.969751] LustreError: 106056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139828 with bad export cookie 7609082143625424001 [ 3212.971691] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3212.983873] LustreError: 106056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3213.311863] Lustre: server umount lustre-MDT0001 complete [ 3227.834762] Lustre: server umount lustre-OST0000 complete [ 3230.495334] Lustre: 93319:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139830/real 1783139830] req@ffff8d1ffadcfb80 x1869757021522304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3230.519602] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.538443] Lustre: Skipped 3 previous similar messages [ 3231.698720] Lustre: server umount lustre-OST0001 complete [ 3237.590276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3249.850358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3260.846879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3272.666864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3294.254694] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3305.264629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.309170] Lustre: lustre-MDT0000: reset Object Index mappings [ 3306.156522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.164948] Lustre: Skipped 1 previous similar message [ 3310.832941] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3319.552464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.598562] Lustre: lustre-MDT0001: reset Object Index mappings [ 3324.619269] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3327.418389] Lustre: 115908:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3334.408935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3340.883375] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3344.273295] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3349.643629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3351.291858] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3351.294729] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3351.401031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3356.134585] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3364.614474] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3368.653070] Lustre: 117786:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3428.125090] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:40:41 (1783140041) [ 3453.414160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3453.426900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.440635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3453.448058] Lustre: Skipped 4 previous similar messages [ 3459.034580] Lustre: server umount lustre-MDT0000 complete [ 3462.511891] LustreError: 114742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140078 with bad export cookie 7609082143625471104 [ 3462.531053] LustreError: 114742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3462.535230] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3467.234403] LustreError: 114755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.244409] LustreError: 114755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 3469.074671] Lustre: server umount lustre-MDT0001 complete [ 3479.417824] Lustre: server umount lustre-OST0000 complete [ 3489.401127] Lustre: server umount lustre-OST0001 complete [ 3497.253215] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 3510.971697] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3550.002273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.245148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3550.280783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3550.342443] Lustre: lustre-MDT0000: new disk, initializing [ 3550.424894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.430678] Lustre: Skipped 3 previous similar messages [ 3550.443655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3555.074766] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3561.809712] Lustre: server umount lustre-MDT0000 complete [ 3569.999483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.225031] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3570.249290] Lustre: 122060:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3570.272152] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3574.448720] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3579.652386] Lustre: server umount lustre-MDT0000 complete [ 3588.789587] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:43:22 (1783140202) [ 3590.617541] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3592.486784] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:43:26 (1783140206) [ 3615.070913] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3629.873875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3635.608302] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3645.434983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3645.614861] Lustre: 124111:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3645.697763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3645.702222] Lustre: Skipped 1 previous similar message [ 3645.791795] Lustre: lustre-MDT0001: new disk, initializing [ 3645.961948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3645.979066] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3650.624912] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3656.025059] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3663.963521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3664.244834] Lustre: lustre-OST0000: new disk, initializing [ 3664.249267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3664.259944] Lustre: 125690:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3665.393954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3665.414866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3665.542536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3672.100772] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3685.522047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3685.900076] Lustre: lustre-OST0001: new disk, initializing [ 3685.909745] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3685.930984] Lustre: 126565:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3687.949802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3687.958758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3688.095666] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3693.344696] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3703.457655] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3707.307905] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4096.551714] Lustre: 132986:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4096.559136] Lustre: 132986:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4096.570188] Lustre: 132986:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4096.579048] Lustre: 132986:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4096.586773] Lustre: 132986:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4096.592929] Lustre: 132986:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4382.043242] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:56:35 (1783140995) [ 4398.560522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4398.567928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.583601] Lustre: Skipped 5 previous similar messages [ 4398.589159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4398.594350] Lustre: Skipped 7 previous similar messages [ 4402.047274] Lustre: server umount lustre-MDT0000 complete [ 4404.706805] LustreError: 124122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.728908] LustreError: 124122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4405.370409] LustreError: 124103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141021 with bad export cookie 7609082143625541433 [ 4405.372072] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4405.379718] LustreError: 124103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4409.826941] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.835885] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4409.843876] Lustre: Skipped 4 previous similar messages [ 4409.851516] Lustre: Skipped 5 previous similar messages [ 4411.950199] Lustre: server umount lustre-MDT0001 complete [ 4421.636122] Lustre: server umount lustre-OST0000 complete [ 4430.853742] Lustre: server umount lustre-OST0001 complete [ 4435.994810] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 4442.586711] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4468.356431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4468.499592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4468.523897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4468.578743] Lustre: lustre-MDT0000: new disk, initializing [ 4468.632621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4468.637028] Lustre: Skipped 5 previous similar messages [ 4468.651043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4471.896793] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4479.928859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.023264] Lustre: 136038:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4480.081135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4480.084962] Lustre: Skipped 1 previous similar message [ 4480.150801] Lustre: lustre-MDT0001: new disk, initializing [ 4480.262055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4480.275546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4483.509495] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4490.838271] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4495.260703] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4500.655649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4500.906555] Lustre: lustre-OST0000: new disk, initializing [ 4500.911451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4500.916565] Lustre: 138007:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4501.008875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4501.012138] Lustre: Skipped 1 previous similar message [ 4502.693313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4502.702499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4502.822611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4506.576645] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4514.956876] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4524.979957] Lustre: lustre-MDD0000: changelog on [ 4527.542412] Lustre: lustre-MDD0001: changelog on [ 6583.528457] Lustre: lustre-MDD0001: changelog off [ 6584.955846] Lustre: 136046:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6585.638434] Lustre: lustre-MDD0000: changelog off [ 6591.968747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6591.974740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6591.977972] LustreError: Skipped 1 previous similar message [ 6591.979773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6591.991841] Lustre: Skipped 1 previous similar message [ 6592.003912] Lustre: Skipped 3 previous similar messages [ 6597.090839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6597.098436] Lustre: Skipped 1 previous similar message [ 6602.213966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6602.218717] Lustre: Skipped 1 previous similar message [ 6605.279290] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6605.418805] Lustre: server umount lustre-OST0000 complete [ 6610.401365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6610.410395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.420189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6612.450838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6614.469863] Lustre: server umount lustre-MDT0000 complete [ 6617.446660] LustreError: 136032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143233 with bad export cookie 7609082143626366544 [ 6617.457975] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6617.686564] Lustre: server umount lustre-MDT0001 complete [ 6625.698460] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6628.040831] Key type lgssc unregistered [ 6628.313192] LNet: 147179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6628.321720] LNetError: 147179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6628.339630] LNet: Removed LNI 192.168.204.148@tcp [ 6629.063347] Key type .llcrypt unregistered [ 6629.066887] Key type ._llcrypt unregistered [ 6644.277410] Key type ._llcrypt registered [ 6644.279523] Key type .llcrypt registered [ 6644.372153] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 6656.274960] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6657.118671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6657.169345] alg: No test for adler32 (adler32-zlib) [ 6658.124309] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 6658.290979] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6659.967313] Key type lgssc registered [ 6660.889151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6688.834910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6688.857947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.037646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6690.075708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6690.148111] Lustre: lustre-MDT0000: new disk, initializing [ 6690.268714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6690.297084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6693.784360] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6701.985741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6702.088821] Lustre: 150355:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6702.171950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6702.181830] Lustre: Skipped 1 previous similar message [ 6702.270749] Lustre: lustre-MDT0001: new disk, initializing [ 6702.356963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6702.388244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6702.404240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6705.844471] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6713.375823] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6717.369444] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6722.454387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6722.723105] Lustre: lustre-OST0000: new disk, initializing [ 6722.726137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6722.732391] Lustre: 152325:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6722.809842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.743917] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6732.825370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6732.834511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6732.912470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6733.142421] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6736.894451] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_module obdecho/obdecho [ 6738.228830] LustreError: 153327:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6738.239258] LustreError: 153327:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6739.547298] LustreError: 153423:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6748.130560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6748.141622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6748.147215] Lustre: Skipped 1 previous similar message [ 6753.077330] Lustre: server umount lustre-MDT0000 complete [ 6755.370714] LustreError: 150349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143371 with bad export cookie 17472233882393679753 [ 6755.374883] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6755.378796] LustreError: 150349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6755.562273] Lustre: server umount lustre-MDT0001 complete [ 6768.544210] Lustre: server umount lustre-OST0000 complete [ 6782.384291] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 6788.888676] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6815.071969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6815.240745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6815.270506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6815.324878] Lustre: lustre-MDT0000: new disk, initializing [ 6815.381895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6815.391504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6818.523381] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6826.222666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6826.323411] Lustre: 157137:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6826.395152] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6826.399739] Lustre: Skipped 1 previous similar message [ 6826.447798] Lustre: lustre-MDT0001: new disk, initializing [ 6826.520474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6826.561467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6826.567161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6829.387826] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6836.255801] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6840.106944] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6844.459475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6844.701232] Lustre: lustre-OST0000: new disk, initializing [ 6844.709388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6844.715271] Lustre: 159101:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6844.792558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6848.799381] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6851.626970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6851.637904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6851.707809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6855.086088] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6857.975700] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6858.122422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6860.925733] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6861.056699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6862.306171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6862.311082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.317297] Lustre: Skipped 2 previous similar messages [ 6862.321657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6866.916627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6866.918399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6866.918441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6868.443404] Lustre: server umount lustre-OST0000 complete [ 6872.033274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6872.037851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6872.041236] Lustre: Skipped 1 previous similar message [ 6872.048770] Lustre: Skipped 1 previous similar message [ 6877.153262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6877.159102] Lustre: Skipped 2 previous similar messages [ 6885.343083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6885.430684] Lustre: server umount lustre-MDT0000 complete [ 6886.368425] LustreError: 157148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.383666] LustreError: 157148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6887.569045] LustreError: 157128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143503 with bad export cookie 17472233882393680712 [ 6887.571414] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6887.577679] LustreError: 157128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6887.777057] Lustre: server umount lustre-MDT0001 complete [ 6889.733759] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:38:24 (1783143504) [ 6890.709425] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6891.811611] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:38:26 (1783143506) [ 6896.441563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6896.760441] LustreError: 161513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6896.813189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.401045] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6902.240911] LustreError: 161514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6904.445743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6907.110280] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6910.994146] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6913.711167] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6917.400673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6920.829393] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6924.531221] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6935.009188] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6935.015133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6935.022598] Lustre: Skipped 2 previous similar messages [ 6935.028682] Lustre: Skipped 4 previous similar messages [ 6936.204272] Lustre: server umount lustre-MDT0000 complete [ 6938.495869] LustreError: 161494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143554 with bad export cookie 17472233882393681594 [ 6938.500494] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6938.506913] LustreError: 161494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6940.129476] LustreError: 161509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6944.788865] Lustre: server umount lustre-MDT0001 complete [ 6948.691144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6957.013536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6957.153100] LustreError: 165348:0:(llog.c:1646:llog_backup()) MGC192.168.204.148@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6957.160089] Lustre: 165348:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.148@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6961.093337] LustreError: 165353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6961.104948] LustreError: 165353:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6972.292209] LustreError: 165370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6972.304920] LustreError: 165370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6992.778709] LustreError: 165354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6992.797115] LustreError: 165354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 7018.463390] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7018.469423] Lustre: 165348:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.148@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7027.589036] LustreError: 165370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7027.602799] LustreError: 165370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 7058.996635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7059.001354] Lustre: Skipped 2 previous similar messages [ 7061.625348] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7064.037338] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7066.612064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7083.487990] LustreError: 148580:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ccb3f042 x1869761241791616/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7086.077452] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7089.128773] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7090.391734] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7091.355470] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7094.435771] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7095.304041] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7109.603811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7109.605763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7109.611677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7109.611689] Lustre: Skipped 4 previous similar messages [ 7109.614982] Lustre: Skipped 2 previous similar messages [ 7120.351134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7120.446345] Lustre: server umount lustre-MDT0000 complete [ 7122.614209] LustreError: 161494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143738 with bad export cookie 17472233882393688055 [ 7122.614768] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7122.623866] LustreError: 161494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7122.812719] Lustre: server umount lustre-MDT0001 complete [ 7134.313901] Lustre: server umount lustre-OST0000 complete [ 7146.418799] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 7152.314247] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7171.999720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7172.119931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7172.141519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7172.194778] Lustre: lustre-MDT0000: new disk, initializing [ 7172.239105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.243078] Lustre: Skipped 1 previous similar message [ 7172.251806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7174.687324] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7181.197915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7181.259746] Lustre: 170431:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7181.400749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7183.552925] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7189.029964] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7192.152974] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7196.039346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7196.208511] Lustre: 172398:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7199.236286] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7199.590218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7199.639299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7204.326611] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7206.423677] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7206.552326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7208.827978] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7208.932895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7209.958514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7209.964627] Lustre: Skipped 1 previous similar message [ 7209.967035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7209.967646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7209.973404] Lustre: Skipped 9 previous similar messages [ 7224.287130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7224.375260] Lustre: server umount lustre-OST0000 complete [ 7227.360375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7227.363869] LustreError: Skipped 1 previous similar message [ 7232.441629] Lustre: server umount lustre-MDT0000 complete [ 7234.390680] LustreError: 170422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143850 with bad export cookie 17472233882393704883 [ 7234.391888] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7234.397722] LustreError: 170422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7234.591902] Lustre: server umount lustre-MDT0001 complete [ 7239.188960] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:44:13 (1783143853) [ 7242.885591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.138323] LustreError: 174806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.149368] LustreError: 174806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 7245.151865] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7249.663645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.099720] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7255.896469] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7258.848477] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7262.286414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7265.435236] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7269.303048] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7278.870106] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:44:53 (1783143893) [ 7284.192942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7284.199588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7284.216203] Lustre: Skipped 3 previous similar messages [ 7284.225682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7284.233351] Lustre: Skipped 7 previous similar messages [ 7287.092414] Lustre: server umount lustre-OST0000 complete [ 7295.405848] Lustre: server umount lustre-MDT0000 complete [ 7297.398692] LustreError: 174786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143913 with bad export cookie 17472233882393705765 [ 7297.400437] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7297.407805] LustreError: 174786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7297.589941] Lustre: server umount lustre-MDT0001 complete [ 7303.664486] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7305.413077] Key type lgssc unregistered [ 7305.599440] LNet: 178531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7305.605144] LNetError: 178531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7305.622621] LNet: Removed LNI 192.168.204.148@tcp [ 7306.152221] Key type .llcrypt unregistered [ 7306.153816] Key type ._llcrypt unregistered [ 7317.112407] Key type ._llcrypt registered [ 7317.113940] Key type .llcrypt registered [ 7317.171664] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7317.677752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7317.686536] alg: No test for adler32 (adler32-zlib) [ 7318.591556] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 7318.729665] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 7320.335453] Key type lgssc registered [ 7320.876654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7324.717849] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7324.728406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7340.192169] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7345.311738] LustreError: 179751:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 7350.431625] LustreError: 179751:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7350.440161] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7350.443869] LustreError: MGC192.168.204.148@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7350.451302] LustreError: 179751:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7350.456651] LustreError: 179751:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7350.464261] LustreError: 179751:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7350.495716] Lustre: server umount lustre-OST0000 complete [ 7350.498944] LustreError: 179751:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7355.142048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.360031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7358.742711] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7363.882830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.109438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7366.483718] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7370.581193] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7373.779561] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7377.380197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.490518] LustreError: 182064:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7377.496949] Lustre: 182064:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.148@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7377.576049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7380.575135] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7384.345028] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7387.105370] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7387.112781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7387.113031] Lustre: Skipped 2 previous similar messages [ 7387.122694] Lustre: Skipped 2 previous similar messages [ 7388.129863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7388.135840] Lustre: Skipped 2 previous similar messages [ 7391.542144] Lustre: server umount lustre-MDT0000 complete [ 7393.249121] LustreError: 180250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7393.264735] LustreError: 180250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7393.534938] LustreError: 180230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144009 with bad export cookie 14033128666965768638 [ 7393.541305] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7393.543283] LustreError: 180230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7394.271277] LustreError: 180980:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000dc20f7a6 x1869761933761152/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7394.370853] Lustre: server umount lustre-MDT0001 complete [ 7406.526332] Lustre: server umount lustre-OST0000 complete [ 7412.357948] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 7418.019992] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7438.446566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.592809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7438.613798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7438.679966] Lustre: lustre-MDT0000: new disk, initializing [ 7438.734852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.747966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7441.044337] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7447.435234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.521686] Lustre: 185586:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7447.578638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7447.581108] Lustre: Skipped 1 previous similar message [ 7447.629944] Lustre: lustre-MDT0001: new disk, initializing [ 7447.707645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7447.730181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7447.743862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7450.236167] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7455.740387] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7458.683559] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7462.344402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7462.496493] Lustre: lustre-OST0000: new disk, initializing [ 7462.499313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7462.502390] Lustre: 187556:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7462.552305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7465.792074] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7471.178862] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7472.139251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7472.148253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7472.208956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7473.353631] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7473.450382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7475.702724] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7475.808863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7477.218048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7477.218311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.222106] LustreError: Skipped 1 previous similar message [ 7477.223227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7477.226981] Lustre: Skipped 1 previous similar message [ 7477.237484] Lustre: Skipped 1 previous similar message [ 7478.899886] Lustre: server umount lustre-OST0000 complete [ 7482.340186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7482.345931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7482.348441] Lustre: Skipped 1 previous similar message [ 7487.220572] Lustre: server umount lustre-MDT0000 complete [ 7487.465051] LustreError: 186956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7487.476395] LustreError: 186956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7488.975967] LustreError: 187555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144104 with bad export cookie 14033128666965769331 [ 7488.980461] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7488.982249] LustreError: 187555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7489.114940] Lustre: server umount lustre-MDT0001 complete [ 7493.405646] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:48:28 (1783144108) [ 7502.467113] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7503.850723] Key type lgssc unregistered [ 7504.025144] LNet: 190496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7504.031292] LNetError: 190496:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7504.040654] LNet: Removed LNI 192.168.204.148@tcp [ 7504.466746] Key type .llcrypt unregistered [ 7504.469417] Key type ._llcrypt unregistered [ 7513.763974] Key type ._llcrypt registered [ 7513.765612] Key type .llcrypt registered [ 7513.815811] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7514.218207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7514.267181] alg: No test for adler32 (adler32-zlib) [ 7515.157555] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 7515.280019] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 7516.879757] Key type lgssc registered [ 7517.421750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7521.230764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7521.248311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7522.466973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.620532] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7528.416563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7528.606614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7530.665033] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7534.065664] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7536.635313] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7540.789862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7540.906383] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7540.909237] LustreError: 193552:0:(llog.c:1687:llog_backup()) MGC192.168.204.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7540.914765] Lustre: 193552:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.148@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7540.976812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7544.371281] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7548.545159] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7552.482481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7552.484333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7552.490288] Lustre: Skipped 2 previous similar messages [ 7552.493283] Lustre: Skipped 1 previous similar message [ 7553.506604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7553.515028] Lustre: Skipped 3 previous similar messages [ 7558.625061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7558.630624] Lustre: Skipped 2 previous similar messages [ 7563.749436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7563.757368] Lustre: Skipped 2 previous similar messages [ 7564.257148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7564.356379] Lustre: server umount lustre-MDT0000 complete [ 7566.315771] LustreError: 194312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144182 with bad export cookie 5209456802918993621 [ 7566.319274] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7566.324541] LustreError: 194312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7567.839290] LustreError: 192418:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000adadfe15 x1869762140331520/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7567.916409] Lustre: server umount lustre-MDT0001 complete [ 7579.256920] Lustre: server umount lustre-OST0000 complete [ 7583.952875] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 7589.191354] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7608.361225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7608.491735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7608.514397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7608.566450] Lustre: lustre-MDT0000: new disk, initializing [ 7608.610829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7608.624589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7610.960904] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7617.318565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7617.378235] Lustre: 197073:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7617.410186] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7617.413392] Lustre: Skipped 1 previous similar message [ 7617.416227] LustreError: 197854:0:(llog.c:1687:llog_backup()) MGC192.168.204.148@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7617.420787] LustreError: 197854:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7617.424494] Lustre: 197854:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.148@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7617.437301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7617.440827] Lustre: Skipped 1 previous similar message [ 7617.485823] Lustre: lustre-MDT0001: new disk, initializing [ 7617.542439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7617.561676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7617.567386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7619.691461] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7624.965438] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7627.634877] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7631.069216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7631.187969] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7631.189940] Lustre: Skipped 1 previous similar message [ 7631.191817] LustreError: 199014:0:(llog.c:1687:llog_backup()) MGC192.168.204.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7631.196268] LustreError: 199014:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7631.199382] Lustre: 199014:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.148@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7631.204849] Lustre: 199014:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7631.217754] Lustre: lustre-OST0000: new disk, initializing [ 7631.220971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7631.225659] Lustre: 199041:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7631.276403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7634.297847] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7639.549817] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7641.097272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7641.103659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7641.172588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7642.019962] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7642.166766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7644.508441] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7644.623377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7646.179125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.179539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7646.190413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7646.193327] LustreError: Skipped 1 previous similar message [ 7647.779689] Lustre: server umount lustre-OST0000 complete [ 7651.296916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7651.302936] Lustre: Skipped 1 previous similar message [ 7656.416983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7656.424645] Lustre: Skipped 4 previous similar messages [ 7664.096094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7664.188281] Lustre: server umount lustre-MDT0000 complete [ 7666.412771] LustreError: 197064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144282 with bad export cookie 5209456802918994328 [ 7666.416727] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7666.422253] LustreError: 197064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7666.627353] Lustre: server umount lustre-MDT0001 complete [ 7671.817669] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:51:26 (1783144286) [ 7680.688440] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7685.996105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.233672] LustreError: 202163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.244691] LustreError: 202163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7686.280902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7688.591541] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7691.746339] LustreError: 202164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7693.299661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7695.706749] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7697.365690] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7700.878963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7704.205773] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7709.117782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7709.202236] Lustre: lustre-OST0001: new disk, initializing [ 7709.205397] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7709.211708] Lustre: 204469:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7711.589985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7711.595700] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7711.654605] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7712.510532] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7719.680684] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7723.311562] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7724.572401] Lustre: DEBUG MARKER: ADD OST3 [ 7727.409839] loop: module loaded [ 7727.435089] loop0: detected capacity change from 0 to 204800000 [ 7727.532899] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7727.541585] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7727.545829] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7730.856751] loop0: detected capacity change from 0 to 204800000 [ 7733.046185] Lustre: DEBUG MARKER: STOP OST3 [ 7733.694676] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7733.793608] Lustre: lustre-OST0003: new disk, initializing [ 7733.797461] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7733.803396] Lustre: 206303:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7733.887858] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7733.891793] Lustre: Skipped 3 previous similar messages [ 7735.612220] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7735.676858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7735.687444] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7735.716811] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7736.106639] Lustre: Failing over lustre-OST0003 [ 7736.177364] Lustre: server umount lustre-OST0003 complete [ 7736.998340] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7737.631261] LustreError: 206612:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7740.315314] LustreError: 203648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7740.897763] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7740.903539] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.908700] Lustre: Skipped 2 previous similar messages [ 7742.257657] Lustre: DEBUG MARKER: START OST3 again [ 7745.424158] LustreError: 203648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.436218] LustreError: 203648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7745.807841] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7745.979651] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7747.663433] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7747.833812] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7747.833921] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7748.988097] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7764.587591] Lustre: Failing over lustre-OST0003 [ 7764.644418] Lustre: server umount lustre-OST0003 complete [ 7765.912071] LustreError: 203646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7766.504376] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7776.752636] LustreError: 203646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.763928] LustreError: 203646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7780.832230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7780.836678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7780.846652] Lustre: Skipped 1 previous similar message [ 7780.850051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7780.854437] Lustre: Skipped 2 previous similar messages [ 7784.970731] Lustre: server umount lustre-MDT0000 complete [ 7786.730868] LustreError: 202145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144402 with bad export cookie 5209456802918995210 [ 7786.738159] LustreError: 202145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7786.740356] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7796.193757] LustreError: 204412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7796.201605] LustreError: 204412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 7801.312359] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7801.424932] Lustre: server umount lustre-MDT0001 complete [ 7803.182250] Lustre: server umount lustre-OST0000 complete [ 7804.935041] Lustre: server umount lustre-OST0001 complete [ 7807.914518] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 7811.522663] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7828.584655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.683227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7828.700860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7828.745272] Lustre: lustre-MDT0000: new disk, initializing [ 7828.783913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.789995] Lustre: Skipped 1 previous similar message [ 7828.801645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7830.660544] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7835.294035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.348911] Lustre: 211768:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7835.485732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7837.236655] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7841.597382] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7843.865711] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7846.947451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7847.097987] Lustre: lustre-OST0000: new disk, initializing [ 7847.100858] Lustre: Skipped 1 previous similar message [ 7847.104512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7847.108530] Lustre: Skipped 2 previous similar messages [ 7847.112964] Lustre: 213735:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7849.030145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7849.035072] Lustre: Skipped 1 previous similar message [ 7849.037878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7849.077079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7849.823163] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7854.272259] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7856.188353] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7856.308845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7858.232299] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7858.339761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7859.169683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7859.170760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7859.176886] Lustre: Skipped 6 previous similar messages [ 7859.179505] Lustre: Skipped 12 previous similar messages [ 7865.237814] Lustre: server umount lustre-OST0000 complete [ 7881.183256] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7881.246647] Lustre: server umount lustre-MDT0000 complete [ 7882.771201] LustreError: 211760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144498 with bad export cookie 5209456802918997506 [ 7882.773737] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7882.775756] LustreError: 211760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7882.905259] Lustre: server umount lustre-MDT0001 complete [ 7884.395279] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:54:59 (1783144499) [ 7890.895917] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 7894.993838] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7911.135276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7911.264483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7911.279160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7911.320152] Lustre: lustre-MDT0000: new disk, initializing [ 7911.350480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7913.154548] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7918.299888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7918.356058] Lustre: 218337:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7918.497660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7920.317259] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7924.774505] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7927.213345] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7930.187416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7930.326313] Lustre: 220303:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7931.454371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7932.912871] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7937.572212] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7939.507692] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7939.609541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7941.439601] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7941.531291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7946.723074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7946.723762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7946.724578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7946.724583] Lustre: Skipped 8 previous similar messages [ 7946.730324] Lustre: Skipped 3 previous similar messages [ 7946.734080] LustreError: Skipped 1 previous similar message [ 7948.527507] Lustre: server umount lustre-OST0000 complete [ 7956.541638] Lustre: server umount lustre-MDT0000 complete [ 7956.961505] LustreError: 218349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7956.970584] LustreError: 218349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7958.263676] LustreError: 219131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144574 with bad export cookie 5209456802918998388 [ 7958.270619] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7958.273143] LustreError: 219131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7958.444619] Lustre: server umount lustre-MDT0001 complete [ 7963.297513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7963.608630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7963.612495] Lustre: Skipped 5 previous similar messages [ 7965.937761] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7970.608827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.160817] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7976.950093] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7979.680272] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7983.434119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7986.755253] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7990.446637] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7998.945664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8004.084285] Lustre: server umount lustre-OST0000 complete [ 8011.234577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8011.244542] Lustre: Skipped 5 previous similar messages [ 8012.361410] Lustre: server umount lustre-MDT0000 complete [ 8014.297951] LustreError: 222498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144630 with bad export cookie 5209456802918999270 [ 8014.302711] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8014.304684] LustreError: 222498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8014.471491] Lustre: server umount lustre-MDT0001 complete [ 8123.418264] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 8124.638499] Key type lgssc unregistered [ 8124.802723] LNet: 226176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8124.806797] LNetError: 226176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8124.819670] LNet: Removed LNI 192.168.204.148@tcp [ 8125.209785] Key type .llcrypt unregistered [ 8125.211526] Key type ._llcrypt unregistered [ 8135.285467] Key type ._llcrypt registered [ 8135.287443] Key type .llcrypt registered [ 8135.334979] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 8142.466273] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8142.830044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8142.879396] alg: No test for adler32 (adler32-zlib) [ 8143.753970] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8143.860641] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 8145.447166] Key type lgssc registered [ 8145.861049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8159.287885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8159.293972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8160.394926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8160.409983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8160.450819] Lustre: lustre-MDT0000: new disk, initializing [ 8160.481480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8160.488098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8162.083337] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8167.103422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8167.167561] Lustre: 229503:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8167.202662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8167.205350] Lustre: Skipped 1 previous similar message [ 8167.248340] Lustre: lustre-MDT0001: new disk, initializing [ 8167.297486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8167.316267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8167.321346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8168.912470] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8173.302255] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8175.490842] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8178.280361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8178.396821] Lustre: lustre-OST0000: new disk, initializing [ 8178.398856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8178.402238] Lustre: 231470:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8178.435651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.793135] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8185.022699] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8186.365323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8186.369807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8186.407326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8186.589638] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8186.675327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8188.145287] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8188.219732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8191.457078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8191.459253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8191.460023] LustreError: Skipped 1 previous similar message [ 8191.465718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8191.469273] Lustre: Skipped 1 previous similar message [ 8195.030737] Lustre: server umount lustre-OST0000 complete [ 8196.576216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8196.576908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8196.581140] Lustre: Skipped 2 previous similar messages [ 8196.585780] Lustre: Skipped 1 previous similar message [ 8201.696680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8201.701561] Lustre: Skipped 1 previous similar message [ 8202.657941] Lustre: server umount lustre-MDT0000 complete [ 8204.137336] LustreError: 231469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144820 with bad export cookie 12824685586108629462 [ 8204.139160] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8204.143592] LustreError: 231469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8204.268589] Lustre: server umount lustre-MDT0001 complete [ 8212.035872] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8215.884314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.059989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.599865] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8220.863392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.010603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8222.680207] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8223.820454] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8226.396633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8228.954159] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8232.031638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8232.099354] Lustre: lustre-OST0001: new disk, initializing [ 8232.101774] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8232.105077] Lustre: 236702:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8232.140267] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8232.143615] Lustre: Skipped 1 previous similar message [ 8234.243520] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8236.548485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8236.552536] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8236.577120] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8244.268279] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8252.374915] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8257.080183] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:01:11 (1783144871) [ 8261.090964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8261.096696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8261.096828] Lustre: Skipped 2 previous similar messages [ 8261.099405] Lustre: Skipped 1 previous similar message [ 8264.050169] Lustre: server umount lustre-MDT0000 complete [ 8265.582247] LustreError: 234376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144881 with bad export cookie 12824685586108630330 [ 8265.585033] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8265.587611] LustreError: 234376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8265.738647] Lustre: server umount lustre-MDT0001 complete [ 8277.769768] Lustre: server umount lustre-OST0000 complete [ 8289.763950] Lustre: server umount lustre-OST0001 complete [ 8292.255320] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 8296.211599] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8311.191562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8311.283333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8311.294319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8311.334760] Lustre: lustre-MDT0000: new disk, initializing [ 8311.357989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8311.368212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8312.920359] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8317.621223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8317.684579] Lustre: 240651:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8317.814698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8319.372401] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8323.557220] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8325.771568] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8328.350835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8328.452679] Lustre: lustre-OST0000: new disk, initializing [ 8328.454767] Lustre: Skipped 1 previous similar message [ 8328.456641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8328.459753] Lustre: Skipped 2 previous similar messages [ 8328.462505] Lustre: 242617:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8330.111238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8330.115315] Lustre: Skipped 1 previous similar message [ 8330.118573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8330.161453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8330.666553] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8334.672527] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8336.267548] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8336.351523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8337.895701] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8337.984940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8340.448288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8340.449969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.456422] LustreError: Skipped 1 previous similar message [ 8340.461369] Lustre: Skipped 1 previous similar message [ 8340.463198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8340.467951] Lustre: Skipped 1 previous similar message [ 8350.691662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8350.694619] Lustre: Skipped 3 previous similar messages [ 8353.247137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8353.312498] Lustre: server umount lustre-OST0000 complete [ 8355.809975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8355.817317] Lustre: Skipped 2 previous similar messages [ 8361.062229] Lustre: server umount lustre-MDT0000 complete [ 8362.486247] LustreError: 240642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144978 with bad export cookie 12824685586108632451 [ 8362.488903] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8362.493275] LustreError: 240642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8362.630995] Lustre: server umount lustre-MDT0001 complete [ 8365.970636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8366.183316] LustreError: 244830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8366.191816] LustreError: 244830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8366.218864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8366.221792] Lustre: Skipped 2 previous similar messages [ 8367.872502] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8371.667039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8371.680257] LustreError: 244831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.624704] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8376.658776] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8378.760831] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8381.450649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8384.271688] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8387.569145] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8530.402270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8530.402535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8530.406788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8530.410677] Lustre: Skipped 1 previous similar message [ 8530.413108] Lustre: Skipped 4 previous similar messages [ 8544.223226] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8544.289474] Lustre: server umount lustre-OST0000 complete [ 8546.271941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8550.882404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.892243] Lustre: Skipped 1 previous similar message [ 8560.095472] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8560.162838] Lustre: server umount lustre-MDT0000 complete [ 8561.120235] LustreError: 244830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8561.127463] LustreError: 244830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8561.647504] LustreError: 244812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145177 with bad export cookie 12824685586108633333 [ 8561.650911] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8561.652802] LustreError: 244812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8561.778105] Lustre: server umount lustre-MDT0001 complete [ 8651.425066] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 8652.495317] Key type lgssc unregistered [ 8652.630348] LNet: 248437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8652.633282] LNetError: 248437:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8652.642324] LNet: Removed LNI 192.168.204.148@tcp [ 8652.975123] Key type .llcrypt unregistered [ 8652.976450] Key type ._llcrypt unregistered [ 8661.920300] Key type ._llcrypt registered [ 8661.921741] Key type .llcrypt registered [ 8661.963770] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 8668.806720] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8669.130389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8669.157063] alg: No test for adler32 (adler32-zlib) [ 8670.024068] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8670.122869] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 8671.711166] Key type lgssc registered [ 8672.170495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8690.521942] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8694.947235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8694.961966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.055399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8696.068198] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8696.107559] Lustre: lustre-MDT0000: new disk, initializing [ 8696.138307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8696.146390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8697.760060] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8702.700579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.758968] Lustre: 252476:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8702.790604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8702.793432] Lustre: Skipped 1 previous similar message [ 8702.840254] Lustre: lustre-MDT0001: new disk, initializing [ 8702.884409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8702.902810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8702.907962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8704.562754] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8707.017558] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8709.462955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8709.563231] Lustre: lustre-OST0000: new disk, initializing [ 8709.565433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8709.569345] Lustre: 254107:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8709.599582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8711.035805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8711.040618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8711.085157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8711.784513] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8716.305289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8716.372664] Lustre: lustre-OST0001: new disk, initializing [ 8716.375092] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8716.379039] Lustre: 254981:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8716.420570] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8716.568531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8716.573039] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8716.611391] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8718.554565] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8724.171884] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8731.009550] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8737.253346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8737.254422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8737.258835] Lustre: Skipped 2 previous similar messages [ 8737.261645] Lustre: Skipped 1 previous similar message [ 8740.272799] Lustre: server umount lustre-MDT0000 complete [ 8741.603355] LustreError: 252470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145357 with bad export cookie 962455077852075155 [ 8741.607117] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8741.607350] LustreError: 252470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8741.732460] Lustre: server umount lustre-MDT0001 complete [ 8753.429232] Lustre: server umount lustre-OST0000 complete [ 8765.090522] Lustre: server umount lustre-OST0001 complete [ 8770.086236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8773.832644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8777.107490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8780.884201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8788.190244] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8792.183313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8792.194707] Lustre: lustre-MDT0000: reset Object Index mappings [ 8792.406292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8794.009842] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8797.326966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.336297] Lustre: lustre-MDT0001: reset Object Index mappings [ 8799.061565] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8800.227818] Lustre: 261122:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8802.870820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8803.027606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8803.030650] Lustre: Skipped 1 previous similar message [ 8805.330260] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8807.140563] LustreError: 261476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8807.148744] LustreError: 261476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8808.619705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8810.974192] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8817.128536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8817.130274] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8824.927883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8826.284501] Lustre: 263013:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8828.383968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8828.387778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8828.392446] Lustre: Skipped 1 previous similar message [ 8828.394996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8828.397207] Lustre: Skipped 1 previous similar message [ 8832.483914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8832.485061] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8832.486578] Lustre: Skipped 1 previous similar message [ 8832.492628] Lustre: Skipped 2 previous similar messages [ 8833.693659] Lustre: server umount lustre-MDT0000 complete [ 8835.064252] LustreError: 259960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145450 with bad export cookie 962455077852081931 [ 8835.068162] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8835.068779] LustreError: 259960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8835.200660] Lustre: server umount lustre-MDT0001 complete [ 8846.855734] Lustre: server umount lustre-OST0000 complete [ 8858.531200] Lustre: server umount lustre-OST0001 complete [ 8864.886419] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:11:19 (1783145479) [ 8870.752650] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 8874.295831] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8887.668053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8887.755033] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8887.766835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8887.803208] Lustre: lustre-MDT0000: new disk, initializing [ 8887.827148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8887.829338] Lustre: Skipped 1 previous similar message [ 8887.835405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8889.203283] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8893.641928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8893.688639] Lustre: 267263:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8893.802693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8895.247864] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8899.124303] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8901.103212] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8903.458840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8903.574557] Lustre: lustre-OST0000: new disk, initializing [ 8903.576849] Lustre: Skipped 1 previous similar message [ 8903.579648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8903.582854] Lustre: Skipped 2 previous similar messages [ 8903.585927] Lustre: 269231:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8904.888857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8904.892214] Lustre: Skipped 1 previous similar message [ 8904.894691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8904.932061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8905.739825] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8909.682258] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8911.251914] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8911.328862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8912.907542] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8912.983428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8915.425028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8915.427033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8915.428455] LustreError: Skipped 1 previous similar message [ 8915.434987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8915.438411] Lustre: Skipped 2 previous similar messages [ 8919.771251] Lustre: server umount lustre-OST0000 complete [ 8924.640289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8924.644188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8924.649401] Lustre: Skipped 1 previous similar message [ 8924.651723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8927.322099] Lustre: server umount lustre-MDT0000 complete [ 8928.671465] LustreError: 268067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145544 with bad export cookie 962455077852083996 [ 8928.674765] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8928.676334] LustreError: 268067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8928.804571] Lustre: server umount lustre-MDT0001 complete [ 8935.800907] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8939.569974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8939.703041] LustreError: 272161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8939.714399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8939.716757] Lustre: Skipped 2 previous similar messages [ 8941.199420] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8944.431266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.115475] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8947.249865] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8949.821872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8952.206880] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8955.567624] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8955.638968] Lustre: lustre-OST0001: new disk, initializing [ 8955.641966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8955.646408] Lustre: 274469:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8956.997413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8957.001869] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8957.035731] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8957.964521] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8962.566090] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8963.899905] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8965.866897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8965.880502] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8969.697036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8969.700457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8969.702115] Lustre: Skipped 4 previous similar messages [ 8969.707883] Lustre: Skipped 4 previous similar messages [ 8973.854408] Lustre: server umount lustre-MDT0000 complete [ 8974.817830] LustreError: 272161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8974.826543] LustreError: 272161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8975.187782] LustreError: 272143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145591 with bad export cookie 962455077852084885 [ 8975.191040] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8975.193156] LustreError: 272143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8975.305991] Lustre: server umount lustre-MDT0001 complete [ 8987.150222] Lustre: server umount lustre-OST0000 complete [ 8999.004133] Lustre: server umount lustre-OST0001 complete [ 9005.382301] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 9009.583839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.745266] LustreError: 277518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9009.773125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9009.776141] Lustre: Skipped 3 previous similar messages [ 9011.447758] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9015.264274] LustreError: 277519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9015.457603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9017.343187] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9018.649354] Lustre: 278657:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9021.245397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9023.664647] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9025.511666] LustreError: 279014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9027.012848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9029.367693] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9034.219633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9034.223483] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9038.314823] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9039.757635] Lustre: 280536:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9041.707981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9041.712324] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9045.176560] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:14:19 (1783145659) [ 9059.811667] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.812657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9059.816815] Lustre: Skipped 2 previous similar messages [ 9059.819114] Lustre: Skipped 1 previous similar message [ 9062.945839] Lustre: server umount lustre-MDT0000 complete [ 9064.309963] LustreError: 279840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145680 with bad export cookie 962455077852091206 [ 9064.314213] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9064.315872] LustreError: 279840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9064.928677] LustreError: 279034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9064.937261] LustreError: 279034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9070.518649] Lustre: server umount lustre-MDT0001 complete [ 9078.015255] Lustre: server umount lustre-OST0000 complete [ 9085.487310] Lustre: server umount lustre-OST0001 complete [ 9088.050804] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 9091.621520] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 9108.233782] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 9111.943077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.038561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9112.053070] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9112.094891] Lustre: lustre-MDT0000: new disk, initializing [ 9112.136271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9113.660593] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9117.960441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9118.000723] Lustre: 284766:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9118.103508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9119.449195] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9124.228054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9124.344107] Lustre: 286397:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9125.804441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9126.561060] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9130.968635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9133.062699] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9133.253870] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9137.893452] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9143.599959] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:15:58 (1783145758) [ 9148.387551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9148.388444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9148.391867] Lustre: Skipped 4 previous similar messages [ 9148.396009] Lustre: Skipped 7 previous similar messages [ 9150.507926] Lustre: server umount lustre-MDT0000 complete [ 9151.768181] LustreError: 284759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145767 with bad export cookie 962455077852099081 [ 9151.771921] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9151.772117] LustreError: 284759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9151.879542] Lustre: server umount lustre-MDT0001 complete [ 9163.716617] Lustre: server umount lustre-OST0000 complete [ 9174.239854] Lustre: server umount lustre-OST0001 complete [ 9177.162778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.321648] LustreError: 289513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9177.328342] LustreError: 289513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9177.352774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9177.355181] Lustre: Skipped 7 previous similar messages [ 9178.711317] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9181.941370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9183.738251] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9186.613750] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9188.592862] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9191.156397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9193.443338] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9195.999488] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9213.226865] loop1: detected capacity change from 0 to 204800000 [ 9213.309073] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9213.315939] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9213.321625] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9215.377411] loop1: detected capacity change from 0 to 204800000 [ 9217.417022] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9217.453859] LustreError: 289502:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs28629: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9217.461124] LustreError: 289502:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs28629 [ 9217.463824] LustreError: 289502:0:(mgs_handler.c:517:mgs_target_reg()) fs28629-MDT0001: Error checking target index: rc = -13 [ 9217.469061] LustreError: fs28629-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9217.475579] LustreError: 293250:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9217.478298] LustreError: 293250:0:(tgt_mount.c:2129:server_put_super()) no obd fs28629-MDT0001 [ 9217.481501] LustreError: 293250:0:(tgt_mount.c:116:server_deregister_mount()) fs28629-MDT0001 not registered [ 9217.497159] Lustre: server umount fs28629-MDT0001 complete [ 9217.499058] LustreError: 293250:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9220.832326] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9220.869090] Lustre: Setting parameter fs28629-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs28629-MDT0001 [ 9220.873639] Lustre: Skipped 2 previous similar messages [ 9220.894365] Lustre: srv-fs28629-MDT0001: No data found on store. Initialize space. [ 9220.896735] Lustre: Skipped 4 previous similar messages [ 9220.939756] Lustre: fs28629-MDT0001: new disk, initializing [ 9220.942187] Lustre: Skipped 3 previous similar messages [ 9220.997794] Lustre: srv-fs28629-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9221.001973] Lustre: 293716:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28629-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9221.006627] LustreError: 293716:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28629-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9222.636837] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9225.714255] Lustre: server umount fs28629-MDT0001 complete [ 9230.376862] loop1: detected capacity change from 0 to 204800000 [ 9232.420559] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9232.541252] Lustre: 295281:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28629-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9232.545051] LustreError: 295281:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28629-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9234.103844] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9235.991449] Lustre: server umount fs28629-MDT0001 complete [ 9242.499947] LustreError: 291341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9242.505858] LustreError: 291341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 9242.509403] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 9243.619525] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9243.623822] Lustre: Skipped 3 previous similar messages [ 9243.625813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9245.656357] Lustre: server umount lustre-MDT0000 complete [ 9248.782031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9248.826196] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9250.557757] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9253.591243] loop1: detected capacity change from 0 to 204800000 [ 9254.380140] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9255.869621] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9256.026610] Lustre: 297723:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28629-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9256.030905] LustreError: 297723:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28629-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9257.642129] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9259.502920] Lustre: server umount fs28629-MDT0001 complete [ 9268.191794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9273.107853] Lustre: server umount lustre-OST0000 complete [ 9280.537317] Lustre: server umount lustre-MDT0000 complete [ 9281.760125] LustreError: 295772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145897 with bad export cookie 962455077852101755 [ 9281.763719] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9281.764257] LustreError: 295772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9281.871848] Lustre: server umount lustre-MDT0001 complete [ 9285.880110] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 9286.939577] Key type lgssc unregistered [ 9287.075431] LNet: 300061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9287.078327] LNetError: 300061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9287.086372] LNet: Removed LNI 192.168.204.148@tcp [ 9287.407115] Key type .llcrypt unregistered [ 9287.408229] Key type ._llcrypt unregistered [ 9295.699142] Key type ._llcrypt registered [ 9295.700408] Key type .llcrypt registered [ 9295.741243] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 9296.109654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9296.180962] alg: No test for adler32 (adler32-zlib) [ 9297.067209] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 9297.163357] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 9298.751314] Key type lgssc registered [ 9299.223991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9302.312868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9302.319849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9303.483192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9304.920420] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9307.769868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9307.911198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9309.288653] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9311.753887] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9313.567261] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9315.965374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9316.114476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9317.158387] LustreError: 303139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9317.168936] LustreError: 303139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9318.453449] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9321.300208] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9322.466362] LustreError: 303138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.588669] LustreError: 303139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9329.635466] LustreError: 303140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9329.647113] LustreError: 303140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9334.754239] LustreError: 303138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9334.759928] LustreError: 303138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9338.280490] loop1: detected capacity change from 0 to 204800000 [ 9338.351206] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9338.355497] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9338.359914] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9340.228160] loop1: detected capacity change from 0 to 204800000 [ 9342.469252] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9342.535627] Lustre: fs30647-OST0000: new disk, initializing [ 9342.537956] Lustre: srv-fs30647-OST0000: No data found on store. Initialize space. [ 9342.542478] Lustre: 305062:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9342.572788] Lustre: fs30647-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9344.781853] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9344.994757] LustreError: 303290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9345.000949] LustreError: 303290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9348.413840] Lustre: server umount fs30647-OST0000 complete [ 9352.844427] loop1: detected capacity change from 0 to 204800000 [ 9354.827029] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9354.893939] Lustre: fs30647-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9356.874038] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9359.851083] loop2: detected capacity change from 0 to 204800000 [ 9359.927920] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9359.933250] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9359.937979] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9361.725531] loop2: detected capacity change from 0 to 204800000 [ 9362.820397] LustreError: 303818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9362.828626] LustreError: 303818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 9363.662432] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9363.693729] LustreError: fs30647-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9363.697515] LustreError: fs30647-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9363.701353] LustreError: 308100:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9363.703966] LustreError: 308100:0:(tgt_mount.c:2129:server_put_super()) no obd fs30647-OST0001 [ 9363.706445] LustreError: 308100:0:(tgt_mount.c:116:server_deregister_mount()) fs30647-OST0001 not registered [ 9363.716669] Lustre: server umount fs30647-OST0001 complete [ 9363.718768] LustreError: 308100:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9367.324251] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9367.367262] Lustre: fs30647-OST0001: new disk, initializing [ 9367.369194] Lustre: srv-fs30647-OST0001: No data found on store. Initialize space. [ 9367.371763] Lustre: 308651:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9367.398761] Lustre: fs30647-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9369.401853] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9373.041815] Lustre: server umount fs30647-OST0000 complete [ 9376.566380] Lustre: server umount fs30647-OST0001 complete [ 9381.011353] loop1: detected capacity change from 0 to 204800000 [ 9382.989678] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9385.105075] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9387.413651] loop2: detected capacity change from 0 to 204800000 [ 9389.373460] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9389.438397] Lustre: fs30647-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9389.441397] Lustre: Skipped 1 previous similar message [ 9391.536668] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9394.685255] Lustre: server umount fs30647-OST0000 complete [ 9396.194652] LustreError: 303818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9396.201040] LustreError: 303818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 9397.966544] Lustre: server umount fs30647-OST0001 complete [ 9404.384103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9404.387045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9404.391889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9406.433922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9406.435876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9406.437214] LustreError: Skipped 1 previous similar message [ 9406.439057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9411.555419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9411.564643] Lustre: Skipped 2 previous similar messages [ 9416.159077] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9416.221222] Lustre: server umount lustre-OST0000 complete [ 9420.767713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9420.770396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9420.774569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9423.706724] Lustre: server umount lustre-MDT0000 complete [ 9425.030409] LustreError: 308650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146040 with bad export cookie 15972385943104772557 [ 9425.034807] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9425.037899] LustreError: 308650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9425.155546] Lustre: server umount lustre-MDT0001 complete [ 9428.790172] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 9429.870875] Key type lgssc unregistered [ 9430.007368] LNet: 314561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9430.010576] LNetError: 314561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9430.019459] LNet: Removed LNI 192.168.204.148@tcp [ 9430.371302] Key type .llcrypt unregistered [ 9430.372693] Key type ._llcrypt unregistered [ 9440.940975] Key type ._llcrypt registered [ 9440.942051] Key type .llcrypt registered [ 9440.978733] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 9447.372631] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 9447.696502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9447.733075] alg: No test for adler32 (adler32-zlib) [ 9448.599318] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 9448.696144] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 9450.279113] Key type lgssc registered [ 9450.680361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9463.173369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9463.180057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9464.276395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9464.288729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9464.326235] Lustre: lustre-MDT0000: new disk, initializing [ 9464.348851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9464.354669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9465.826859] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9470.266976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9470.313493] Lustre: 318389:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9470.338997] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9470.341226] Lustre: Skipped 1 previous similar message [ 9470.377634] Lustre: lustre-MDT0001: new disk, initializing [ 9470.414961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9470.428741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9470.434850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9471.958568] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9475.951792] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9477.969421] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9480.578060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.702439] Lustre: lustre-OST0000: new disk, initializing [ 9480.705588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9480.710057] Lustre: 320359:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9480.747351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9483.158566] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9487.554652] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9490.430664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9490.435266] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9490.477442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9495.537521] Lustre: Mounted lustre-client [ 9495.883123] Lustre: Unmounted lustre-client [ 9500.643089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9500.643805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9500.647435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9500.649064] LustreError: Skipped 1 previous similar message [ 9500.650718] Lustre: Skipped 1 previous similar message [ 9505.171808] Lustre: server umount lustre-OST0000 complete [ 9510.881116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9510.882904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9510.885723] Lustre: Skipped 2 previous similar messages [ 9512.600393] Lustre: server umount lustre-MDT0000 complete [ 9513.878023] LustreError: 318382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146129 with bad export cookie 2884670899861072072 [ 9513.880733] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9513.881870] LustreError: 318382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9513.997397] Lustre: server umount lustre-MDT0001 complete [ 9518.046331] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 9519.191311] Key type lgssc unregistered [ 9541.855386] LNet: 322516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9541.857575] LNetError: 322516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9618.661300] LNet: Removed LNI 192.168.204.148@tcp [ 9618.956780] Key type .llcrypt unregistered [ 9618.958058] Key type ._llcrypt unregistered [ 9625.004713] Key type ._llcrypt registered [ 9625.005982] Key type .llcrypt registered [ 9625.042665] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 9631.528114] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 9631.834312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9631.848226] alg: No test for adler32 (adler32-zlib) [ 9632.700538] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 9632.786757] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 9634.367155] Key type lgssc registered [ 9634.742137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9644.631414] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:24:19 (1783146259) [ 9646.984481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9646.992350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9648.078616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9648.090062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9648.127294] Lustre: lustre-MDT0000: new disk, initializing [ 9648.150222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9648.156331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9649.640760] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9653.929290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9653.972094] Lustre: 325715:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9653.996849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9653.999782] Lustre: Skipped 1 previous similar message [ 9654.036354] Lustre: lustre-MDT0001: new disk, initializing [ 9654.071991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9654.086153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9654.090765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9655.551549] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9659.286038] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9661.116081] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9663.323059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9663.417081] Lustre: lustre-OST0000: new disk, initializing [ 9663.418695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9663.421467] Lustre: 327683:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9663.450765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9665.382644] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9666.103104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9666.107134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9666.140436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9669.144959] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9676.257280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9676.258159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9676.261556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9676.265395] Lustre: Skipped 1 previous similar message [ 9676.945852] Lustre: server umount lustre-OST0000 complete [ 9679.839930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9679.843098] LustreError: Skipped 1 previous similar message [ 9679.844882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9679.850345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9679.852637] Lustre: Skipped 1 previous similar message [ 9680.344724] Lustre: server umount lustre-MDT0000 complete [ 9681.377300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9681.653288] LustreError: 325708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146297 with bad export cookie 15555414054418449477 [ 9681.656488] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9681.658390] LustreError: 325708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9681.785136] Lustre: server umount lustre-MDT0001 complete [ 9685.677035] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 9686.729755] Key type lgssc unregistered [ 9686.856302] LNet: 329586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9686.859210] LNetError: 329586:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9686.867338] LNet: Removed LNI 192.168.204.148@tcp [ 9687.183878] Key type .llcrypt unregistered [ 9687.185662] Key type ._llcrypt unregistered [ 9695.694644] Key type ._llcrypt registered [ 9695.696273] Key type .llcrypt registered [ 9695.742907] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 9696.232550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9696.240709] alg: No test for adler32 (adler32-zlib) [ 9697.108168] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 9697.212311] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 9698.807230] Key type lgssc registered [ 9699.233821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9702.009806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9702.017889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9703.184702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9704.640724] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9707.712109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9707.836181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9709.244706] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9711.697656] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9713.470065] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9715.850020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9715.983202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9718.029940] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9720.618426] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9734.426433] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9735.036350] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9738.720914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9738.725209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9738.732264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9739.744379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9739.745907] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9741.196095] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 9741.199744] Lustre: Skipped 2 previous similar messages [ 9742.615772] Lustre: server umount lustre-MDT0000 complete [ 9743.941173] LustreError: 330863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146359 with bad export cookie 10762765798387978407 [ 9743.944066] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9743.946591] LustreError: 330863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9744.863867] LustreError: 330833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9744.864095] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9744.869965] LustreError: 330833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9744.874677] Lustre: Skipped 1 previous similar message [ 9744.878844] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9746.307911] LustreError: 332676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.895920] LustreError: 332377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.896192] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9748.904620] Lustre: Skipped 1 previous similar message [ 9750.125037] Lustre: server umount lustre-MDT0001 complete [ 9757.660335] Lustre: server umount lustre-OST0000 complete [ 9760.540941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9761.908660] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9762.879152] Lustre: 335034:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.148@tcp [ 9763.632932] Lustre: server umount MGS complete [ 9766.455351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9766.609066] LustreError: 335604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9766.630780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9768.027590] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9770.937292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9771.052606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9772.342197] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9775.090624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9777.080733] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9779.484948] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9780.019468] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9781.876272] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9782.487205] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9790.945870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9790.947804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9790.949227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9790.954481] Lustre: Skipped 1 previous similar message [ 9793.489340] Lustre: server umount lustre-OST0000 complete [ 9796.064740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9796.070426] Lustre: Skipped 1 previous similar message [ 9809.375139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9809.435688] Lustre: server umount lustre-MDT0000 complete [ 9810.400291] LustreError: 335604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9810.406977] LustreError: 335604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9810.701292] LustreError: 335584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146426 with bad export cookie 10762765798387979233 [ 9810.704389] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9810.705519] LustreError: 335584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9810.812808] Lustre: server umount lustre-MDT0001 complete [ 9814.500182] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 9815.477309] Key type lgssc unregistered [ 9815.608425] LNet: 339038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9815.611106] LNetError: 339038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9815.617300] LNet: Removed LNI 192.168.204.148@tcp [ 9815.938252] Key type .llcrypt unregistered [ 9815.939277] Key type ._llcrypt unregistered [ 9822.185624] Key type ._llcrypt registered [ 9822.187036] Key type .llcrypt registered [ 9822.227428] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 9828.858466] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 9829.216494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9829.252923] alg: No test for adler32 (adler32-zlib) [ 9830.116366] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 9830.212653] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 9831.799142] Key type lgssc registered [ 9832.181627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9844.574765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9844.580936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9845.664355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9845.674345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9845.712527] Lustre: lustre-MDT0000: new disk, initializing [ 9845.735336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9845.742719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9847.109862] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9851.487120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.533101] Lustre: 342171:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9851.560495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9851.563244] Lustre: Skipped 1 previous similar message [ 9851.598476] Lustre: lustre-MDT0001: new disk, initializing [ 9851.632433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9851.648558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9851.652979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9853.047301] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9856.813490] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9858.583280] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9860.976814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9861.098041] Lustre: lustre-OST0000: new disk, initializing [ 9861.099561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9861.102378] Lustre: 344136:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9861.135139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.752074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9861.755248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9861.793485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9863.065987] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9866.683333] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9868.048537] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9868.121387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9869.453801] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9869.525229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9872.356182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9872.357768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9872.358523] LustreError: Skipped 1 previous similar message [ 9872.358668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9872.363364] Lustre: Skipped 1 previous similar message [ 9876.304577] Lustre: server umount lustre-OST0000 complete [ 9882.592514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9882.592813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9882.593316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9882.593319] Lustre: Skipped 1 previous similar message [ 9882.602856] Lustre: Skipped 1 previous similar message [ 9883.682856] Lustre: server umount lustre-MDT0000 complete [ 9884.934182] LustreError: 342162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146500 with bad export cookie 2578903713054620833 [ 9884.936989] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9884.939032] LustreError: 342162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9885.052063] Lustre: server umount lustre-MDT0001 complete [ 9887.780843] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:28:22 (1783146502) [ 9892.592972] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 9895.559738] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 9906.555031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9906.629674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9906.638963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9906.670101] Lustre: lustre-MDT0000: new disk, initializing [ 9906.690389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9906.696574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9907.918836] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9911.977559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9912.022394] Lustre: 348884:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9912.055935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9912.057729] Lustre: Skipped 1 previous similar message [ 9912.090601] Lustre: lustre-MDT0001: new disk, initializing [ 9912.119941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9912.131594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9912.135063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9913.428237] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9917.042375] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9918.810169] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9921.014092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9921.108461] Lustre: lustre-OST0000: new disk, initializing [ 9921.110285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9921.112973] Lustre: 350851:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9921.139877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9923.030868] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 9925.620967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9925.623994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9925.654664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9926.574390] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9935.841049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9935.841325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9935.845216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9935.846891] LustreError: Skipped 1 previous similar message [ 9935.848735] Lustre: Skipped 1 previous similar message [ 9937.360356] Lustre: server umount lustre-OST0000 complete [ 9940.960733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9940.961203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9940.964506] Lustre: Skipped 2 previous similar messages [ 9940.966143] Lustre: Skipped 1 previous similar message [ 9944.731217] Lustre: server umount lustre-MDT0000 complete [ 9945.985759] LustreError: 348876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146561 with bad export cookie 2578903713054621701 [ 9945.989491] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9945.990297] LustreError: 348876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9946.107561] Lustre: server umount lustre-MDT0001 complete [ 9949.584803] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 9950.588686] Key type lgssc unregistered [ 9950.718351] LNet: 352999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9950.720444] LNetError: 352999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9950.727301] LNet: Removed LNI 192.168.204.148@tcp [ 9951.003928] Key type .llcrypt unregistered [ 9951.005234] Key type ._llcrypt unregistered [ 9981.813227] Key type ._llcrypt registered [ 9981.814406] Key type .llcrypt registered [ 9981.846796] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 9981.938183] Key type .llcrypt unregistered [ 9981.939275] Key type ._llcrypt unregistered [10008.901831] Key type ._llcrypt registered [10008.903330] Key type .llcrypt registered [10008.943669] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10009.043474] Key type .llcrypt unregistered [10009.044506] Key type ._llcrypt unregistered [10020.042619] Key type ._llcrypt registered [10020.044078] Key type .llcrypt registered [10020.081320] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10020.185379] Key type .llcrypt unregistered [10020.186683] Key type ._llcrypt unregistered [10031.112863] Key type ._llcrypt registered [10031.113657] Key type .llcrypt registered [10031.154257] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10031.253477] Key type .llcrypt unregistered [10031.254714] Key type ._llcrypt unregistered [10058.607972] Key type ._llcrypt registered [10058.609662] Key type .llcrypt registered [10058.652928] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10058.750279] Key type .llcrypt unregistered [10058.751546] Key type ._llcrypt unregistered [10069.146368] Key type ._llcrypt registered [10069.147607] Key type .llcrypt registered [10069.183372] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10069.274421] Key type .llcrypt unregistered [10069.275977] Key type ._llcrypt unregistered [10079.594415] Key type ._llcrypt registered [10079.595706] Key type .llcrypt registered [10079.634418] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10079.733426] Key type .llcrypt unregistered [10079.735244] Key type ._llcrypt unregistered [10090.276094] Key type ._llcrypt registered [10090.277728] Key type .llcrypt registered [10090.313519] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10090.407361] Key type .llcrypt unregistered [10090.408961] Key type ._llcrypt unregistered [10101.008035] Key type ._llcrypt registered [10101.009125] Key type .llcrypt registered [10101.044981] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10101.141440] Key type .llcrypt unregistered [10101.142528] Key type ._llcrypt unregistered [10112.052428] Key type ._llcrypt registered [10112.053968] Key type .llcrypt registered [10112.092606] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10112.186478] Key type .llcrypt unregistered [10112.188032] Key type ._llcrypt unregistered [10133.516198] Key type ._llcrypt registered [10133.517760] Key type .llcrypt registered [10133.553044] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10133.645429] Key type .llcrypt unregistered [10133.646909] Key type ._llcrypt unregistered [10144.364932] Key type ._llcrypt registered [10144.366538] Key type .llcrypt registered [10144.403385] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10144.493225] Key type .llcrypt unregistered [10144.494271] Key type ._llcrypt unregistered [10155.264578] Key type ._llcrypt registered [10155.265475] Key type .llcrypt registered [10155.305093] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10155.408221] Key type .llcrypt unregistered [10155.409403] Key type ._llcrypt unregistered [10173.593691] Key type ._llcrypt registered [10173.594901] Key type .llcrypt registered [10173.630845] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [10179.606472] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [10179.904317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10179.924077] alg: No test for adler32 (adler32-zlib) [10180.762594] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [10180.849231] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [10182.431158] Key type lgssc registered [10182.752716] Lustre: Echo OBD driver; http://www.lustre.org/ [10193.437104] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10193.441902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10194.517477] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10194.527963] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10194.557161] Lustre: tst250fs-MDT0000: new disk, initializing [10194.578178] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10194.585164] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10195.776301] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [10199.651173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10199.691411] Lustre: 372912:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10199.712941] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10199.714776] Lustre: Skipped 1 previous similar message [10199.745330] Lustre: tst250fs-MDT0001: new disk, initializing [10199.774333] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10199.785276] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10199.788509] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10201.000744] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [10203.196766] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10205.222921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10205.312806] Lustre: tst250fs-OST0000: new disk, initializing [10205.314325] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10205.316584] Lustre: 374545:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10205.342118] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10207.072847] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [10211.020947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10211.069107] Lustre: tst250fs-OST0001: new disk, initializing [10211.070986] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10211.074668] Lustre: 375403:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10211.104533] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10212.827158] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [10212.864922] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10212.867752] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10212.926550] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10212.931042] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10215.334339] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10225.631873] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10225.634347] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10225.638437] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10228.193840] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10228.194340] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10228.196963] Lustre: Skipped 1 previous similar message [10228.201144] Lustre: Skipped 2 previous similar messages [10231.640477] Lustre: server umount tst250fs-MDT0000 complete [10232.854272] LustreError: 372904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146848 with bad export cookie 17354792286876072747 [10232.856608] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10232.858065] LustreError: 372904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10232.962052] Lustre: server umount tst250fs-MDT0001 complete [10244.415144] Lustre: server umount tst250fs-OST0000 complete [10255.767458] Lustre: server umount tst250fs-OST0001 complete [10261.677557] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [10264.313073] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [10274.680696] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:34:49 (1783146889) [10275.226567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10280.404739] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [10283.680745] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [10293.998704] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10031 sec ========= 02:35:08 (1783146908) [10294.575533] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:35:09 (1783146909) === [10295.689217] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:35:10 (1783146910) === [10308.759820] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [10309.734607] Key type lgssc unregistered [10309.853309] LNet: 384513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10309.857120] LNetError: 384513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10309.866328] LNet: Removed LNI 192.168.204.148@tcp [10310.175140] Key type .llcrypt unregistered [10310.176345] Key type ._llcrypt unregistered