[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 507860685 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002343] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007379] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.013032] SELinux: Initializing. [ 0.014058] *** VALIDATE selinux *** [ 0.022353] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027012] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.032073] *** VALIDATE proc *** [ 0.033246] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037204] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043438] debug: unmapping init [mem 0xffffffff8de59000-0xffffffff8de60fff] [ 0.045912] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046719] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048038] ... bit width: 48 [ 0.049022] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051021] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053027] ... event mask: 000000070000000f [ 0.054360] rcu: Hierarchical SRCU implementation. [ 0.056799] smp: Bringing up secondary CPUs ... [ 0.057709] x86: Booting SMP configuration: [ 0.058031] .... node #0, CPUs: #1 #2 #3 [ 0.070259] smp: Brought up 1 node, 4 CPUs [ 0.072013] smpboot: Max logical packages: 1 [ 0.073012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.107064] node 0 deferred pages initialised in 32ms [ 0.110599] devtmpfs: initialized [ 0.111222] x86/mm: Memory block size: 128MB [ 0.114181] gcov: version magic: 0x41383552 [ 0.116112] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.117070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.118477] pinctrl core: initialized pinctrl subsystem [ 0.119305] [ 0.119871] ************************************************************* [ 0.120012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.121013] ** ** [ 0.122010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.123009] ** ** [ 0.124010] ** This means that this kernel is built to expose internal ** [ 0.125011] ** IOMMU data structures, which may compromise security on ** [ 0.126016] ** your system. ** [ 0.127012] ** ** [ 0.128010] ** If you see this message and you are not debugging the ** [ 0.129012] ** kernel, report this immediately to your vendor! ** [ 0.130010] ** ** [ 0.131021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.132011] ************************************************************* [ 0.133791] NET: Registered protocol family 16 [ 0.134603] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.135058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.136060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.137666] cpuidle: using governor menu [ 0.140032] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.141000] PCI: Using configuration type 1 for base access [ 0.143125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.153398] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.155019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.159056] cryptd: max_cpu_qlen set to 1000 [ 0.160229] ACPI: Added _OSI(Module Device) [ 0.162013] ACPI: Added _OSI(Processor Device) [ 0.164017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.166012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.172067] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.179243] ACPI: Interpreter enabled [ 0.180144] ACPI: PM: (supports S0 S3 S4 S5) [ 0.181010] ACPI: Using IOAPIC for interrupt routing [ 0.182131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.183413] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.195388] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.196056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.197022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.198077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.200689] acpiphp: Slot [2] registered [ 0.201211] acpiphp: Slot [5] registered [ 0.202141] acpiphp: Slot [6] registered [ 0.203151] acpiphp: Slot [7] registered [ 0.204172] acpiphp: Slot [8] registered [ 0.205205] acpiphp: Slot [9] registered [ 0.206220] acpiphp: Slot [10] registered [ 0.207141] acpiphp: Slot [3] registered [ 0.208196] acpiphp: Slot [4] registered [ 0.209208] acpiphp: Slot [11] registered [ 0.210488] acpiphp: Slot [12] registered [ 0.211148] acpiphp: Slot [13] registered [ 0.212146] acpiphp: Slot [14] registered [ 0.213184] acpiphp: Slot [15] registered [ 0.214250] acpiphp: Slot [16] registered [ 0.215286] acpiphp: Slot [17] registered [ 0.216247] acpiphp: Slot [18] registered [ 0.217156] acpiphp: Slot [19] registered [ 0.218174] acpiphp: Slot [20] registered [ 0.219145] acpiphp: Slot [21] registered [ 0.220178] acpiphp: Slot [22] registered [ 0.221000] acpiphp: Slot [23] registered [ 0.223190] acpiphp: Slot [24] registered [ 0.225235] acpiphp: Slot [25] registered [ 0.227303] acpiphp: Slot [26] registered [ 0.230180] acpiphp: Slot [27] registered [ 0.232105] acpiphp: Slot [28] registered [ 0.234230] acpiphp: Slot [29] registered [ 0.235220] acpiphp: Slot [30] registered [ 0.237258] acpiphp: Slot [31] registered [ 0.239092] PCI host bridge to bus 0000:00 [ 0.241034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.244026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.248068] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.250030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.253031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.258027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.260290] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.264337] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.268731] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.290017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.296084] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.300050] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.304019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.306027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.309960] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.313852] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.318043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.322037] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.328017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.341012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.346017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.353859] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.364016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.373000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.373000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.382902] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.388000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.390016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.444030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.469519] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.485022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.493016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.507015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.516000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.516000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.521220] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.543017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.576451] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.594018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.606022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.672022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.690249] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.705048] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.721024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.765014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.787090] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.788000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.789412] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.791345] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.794211] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.799131] iommu: Default domain type: Passthrough [ 0.800046] SCSI subsystem initialized [ 0.802113] ACPI: bus type USB registered [ 0.804112] usbcore: registered new interface driver usbfs [ 0.806103] usbcore: registered new interface driver hub [ 0.808065] usbcore: registered new device driver usb [ 0.810224] pps_core: LinuxPPS API ver. 1 registered [ 0.812011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.815106] PTP clock support registered [ 0.818109] EDAC MC: Ver: 3.0.0 [ 0.820084] PCI: Using ACPI for IRQ routing [ 0.823156] NetLabel: Initializing [ 0.824009] NetLabel: domain hash size = 128 [ 0.826011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.829078] NetLabel: unlabeled traffic allowed by default [ 0.831356] vgaarb: loaded [ 0.833637] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.836013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.842239] clocksource: Switched to clocksource kvm-clock [ 0.970131] VFS: Disk quotas dquot_6.6.0 [ 0.971722] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.974431] *** VALIDATE ramfs *** [ 0.978109] *** VALIDATE hugetlbfs *** [ 0.980237] pnp: PnP ACPI init [ 0.983590] pnp: PnP ACPI: found 6 devices [ 1.007824] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.011288] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.014800] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.017532] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.020424] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.023806] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.026870] NET: Registered protocol family 2 [ 1.030637] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.037782] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.041458] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.046585] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.050171] TCP: Hash tables configured (established 65536 bind 65536) [ 1.053482] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.057661] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.060668] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.064728] NET: Registered protocol family 1 [ 1.068207] RPC: Registered named UNIX socket transport module. [ 1.071204] RPC: Registered udp transport module. [ 1.073312] RPC: Registered tcp transport module. [ 1.075432] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.078335] NET: Registered protocol family 44 [ 1.079889] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.082273] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.085131] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.088357] PCI: CLS 0 bytes, default 64 [ 1.090456] Unpacking initramfs... [ 3.159857] debug: unmapping init [mem 0xffff9b97fcc54000-0xffff9b97fffbffff] [ 3.168715] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.172562] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.177720] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.125129] Initialise system trusted keyrings [ 4.128619] Key type blacklist registered [ 4.130458] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.148312] zbud: loaded [ 4.153314] *** VALIDATE nfs *** [ 4.154217] *** VALIDATE nfs4 *** [ 4.155287] pstore: using deflate compression [ 4.158190] Platform Keyring initialized [ 4.256844] NET: Registered protocol family 38 [ 4.258060] Key type asymmetric registered [ 4.259112] Asymmetric key parser 'x509' registered [ 4.262301] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.265456] io scheduler mq-deadline registered [ 4.267327] io scheduler kyber registered [ 4.269986] io scheduler bfq registered [ 4.272152] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.277754] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.281030] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.285084] ACPI: Power Button [PWRF] [ 4.291846] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.302103] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.325299] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.341645] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.376182] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.409376] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.438228] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.443249] Non-volatile memory driver v1.3 [ 4.444798] Linux agpgart interface v0.103 [ 4.481164] virtio_blk virtio1: [vda] 145840 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.484444] vda: detected capacity change from 0 to 74670080 [ 4.503277] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.506194] vdb: detected capacity change from 0 to 1073741824 [ 4.530675] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.534193] vdc: detected capacity change from 0 to 2621440000 [ 4.562194] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.565266] vdd: detected capacity change from 0 to 2621440000 [ 4.607935] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.610855] vde: detected capacity change from 0 to 4294967296 [ 4.629795] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.632044] vdf: detected capacity change from 0 to 4294967296 [ 4.639227] libphy: Fixed MDIO Bus: probed [ 4.644248] usbcore: registered new interface driver usbserial_generic [ 4.646807] usbserial: USB Serial support registered for generic [ 4.649192] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.653359] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.654903] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.657275] mousedev: PS/2 mouse device common for all mice [ 4.661739] rtc_cmos 00:05: RTC can wake from S4 [ 4.665137] rtc_cmos 00:05: registered as rtc0 [ 4.667715] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.670464] intel_pstate: CPU model not supported [ 4.673278] hid: raw HID events driver (C) Jiri Kosina [ 4.675080] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.676712] usbcore: registered new interface driver usbhid [ 4.684531] usbhid: USB HID core driver [ 4.686286] drop_monitor: Initializing network drop monitor service [ 4.688736] Initializing XFRM netlink socket [ 4.691884] NET: Registered protocol family 10 [ 4.695724] Segment Routing with IPv6 [ 4.697329] NET: Registered protocol family 17 [ 4.699372] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.701829] mpls_gso: MPLS GSO support [ 4.711242] RAS: Correctable Errors collector initialized. [ 4.713691] AVX version of gcm_enc/dec engaged. [ 4.715732] AES CTR mode by8 optimization enabled [ 4.717171] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.911581] sched_clock: Marking stable (4911492048, 0)->(6288005045, -1376512997) [ 4.927099] registered taskstats version 1 [ 4.930212] Loading compiled-in X.509 certificates [ 4.933423] zswap: loaded using pool lzo/zbud [ 4.967165] Key type big_key registered [ 4.983250] Key type encrypted registered [ 4.985371] ima: No TPM chip found, activating TPM-bypass! [ 4.990139] ima: Allocated hash algorithm: sha1 [ 4.992964] ima: No architecture policies found [ 4.995352] evm: Initialising EVM extended attributes: [ 4.997440] evm: security.selinux [ 4.998786] evm: security.ima [ 5.000363] evm: security.capability [ 5.002130] evm: HMAC attrs: 0x1 [ 5.004871] rtc_cmos 00:05: setting system clock to 2026-08-10 13:48:55 UTC (1786369735) [ 5.014168] debug: unmapping init [mem 0xffffffff8ee03000-0xffffffff8effffff] [ 5.019840] debug: unmapping init [mem 0xffffffff8db82000-0xffffffff8de58fff] [ 5.033672] Write protecting the kernel read-only data: 28672k [ 5.039646] debug: unmapping init [mem 0xffffffff8c203000-0xffffffff8c3fffff] [ 5.043291] debug: unmapping init [mem 0xffffffff8cb14000-0xffffffff8cbfffff] [ 5.091591] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.100329] systemd[1]: Detected virtualization kvm. [ 5.102237] systemd[1]: Detected architecture x86-64. [ 5.103512] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.132432] systemd[1]: No hostname configured. [ 5.134661] systemd[1]: Set hostname to . [ 5.136802] random: systemd: uninitialized urandom read (16 bytes read) [ 5.139270] systemd[1]: Initializing machine ID from random generator. [ 5.405316] random: systemd: uninitialized urandom read (16 bytes read) [ 5.410538] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.417842] random: systemd: uninitialized urandom read (16 bytes read) [ 5.421324] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.428284] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.679769] device-mapper: uevent: version 1.0.3 [ 6.682770] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] [ 7.994069] random: fast init done Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ 8.772313] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.015217] scsi host0: ata_piix [ 10.127405] scsi host1: ata_piix [ 10.128635] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.130556] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.532613] random: crng init done [ 14.534686] random: 7 urandom warning(s) missed due to ratelimiting [ 16.938764] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.490336] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.484032] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.735213] SELinux: Disabled at runtime. [ 22.834837] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.847197] systemd[1]: Detected virtualization kvm. [ 22.850865] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.195906] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.201697] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.216853] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.223731] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.231205] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.253639] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.265513] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ 25.042126] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ 26.310724] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.394701] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.856297] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.597198] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.739429] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 34.228111] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit)[ 34.607150] NFS: Registering the id_resolver key type [ 34.609186] Key type id_resolver registered [ 34.610816] Key type id_legacy registered [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg339-server login: [ 85.908509] libcfs: loading out-of-tree module taints kernel. [ 85.944393] Key type ._llcrypt registered [ 85.946441] Key type .llcrypt registered [ 86.017607] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 103.600400] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 106.416060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 106.454846] alg: No test for adler32 (adler32-zlib) [ 107.672822] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 108.333638] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 110.031403] Key type lgssc registered [ 111.416560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 126.016438] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 157.496245] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 167.086776] hrtimer: interrupt took 3686519 ns [ 167.862321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 167.878319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.030626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 169.058179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 169.119703] Lustre: lustre-MDT0000: new disk, initializing [ 169.166234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.179825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.738746] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.393356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.472923] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 183.502741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 183.509759] Lustre: Skipped 1 previous similar message [ 183.520084] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 183.587214] Lustre: lustre-MDT0001: new disk, initializing [ 183.643823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.664618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 183.672409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.116598] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.711627] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.518619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.726905] Lustre: lustre-OST0000: new disk, initializing [ 197.730522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.735948] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.776710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.609218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.621882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 201.675031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 202.319629] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.609036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.722687] Lustre: lustre-OST0001: new disk, initializing [ 212.726755] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 212.730831] Lustre: 9521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.788289] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.293252] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.724121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 221.737657] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 221.762478] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 228.096243] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 237.365919] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 242.182854] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing check_logdir /tmp/testlogs/ [ 246.496155] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing yml_node [ 250.701819] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 252.822823] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 254.626637] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 256.210751] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 09:53:06 EDT 2026 [ 269.107938] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 270.231291] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 286.176693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 286.185755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.194097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.223766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.225646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.232037] Lustre: Skipped 1 previous similar message [ 288.247197] Lustre: Skipped 1 previous similar message [ 289.881368] Lustre: server umount lustre-MDT0000 complete [ 292.126764] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370022 with bad export cookie 7876357027582044722 [ 292.128966] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.137023] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 293.345842] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.347378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 293.358394] Lustre: Skipped 2 previous similar messages [ 293.382760] Lustre: Skipped 2 previous similar messages [ 297.376395] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.381785] Lustre: Skipped 2 previous similar messages [ 298.430921] Lustre: server umount lustre-MDT0001 complete [ 301.279793] Lustre: server umount lustre-OST0000 complete [ 303.864785] Lustre: server umount lustre-OST0001 complete [ 307.792934] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 313.920868] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 337.240391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 337.419540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 337.443389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 337.498845] Lustre: lustre-MDT0000: new disk, initializing [ 337.547638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 337.561506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 340.326683] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 347.629653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.738812] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 347.791570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 347.794932] Lustre: Skipped 1 previous similar message [ 347.852555] Lustre: lustre-MDT0001: new disk, initializing [ 347.921917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 347.951402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 347.957789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 350.896246] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 357.879288] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 361.681786] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 366.435147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 366.670359] Lustre: lustre-OST0000: new disk, initializing [ 366.674123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 366.679123] Lustre: 17033:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 366.773184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.666249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 368.673879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 368.773222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 370.861895] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 377.230777] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 380.522877] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 380.659197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 383.290747] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 383.466215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 389.091150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 389.091648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.092905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 389.104518] Lustre: Skipped 1 previous similar message [ 390.782885] Lustre: server umount lustre-OST0000 complete [ 394.208880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 394.209938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.218735] Lustre: Skipped 2 previous similar messages [ 399.290996] Lustre: server umount lustre-MDT0000 complete [ 399.329603] LustreError: 15077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.341533] LustreError: 15077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 401.674258] LustreError: 15057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370132 with bad export cookie 7876357027582046255 [ 401.679980] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 401.684300] LustreError: 15057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 401.860857] Lustre: server umount lustre-MDT0001 complete [ 405.382431] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:55:35 (1786370135) [ 416.047106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.355422] LustreError: 19907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.409332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.958787] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 421.857032] LustreError: 19908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 423.694176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.430643] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 430.955461] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 434.843874] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 436.189819] Lustre: Failing over lustre-MDT0000 [ 436.378833] Lustre: server umount lustre-MDT0000 complete [ 439.264987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 439.265818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 439.270115] LustreError: 19904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 439.275524] Lustre: Skipped 1 previous similar message [ 444.385345] LustreError: 20448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.394689] LustreError: 20448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 451.783485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.857136] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 454.613985] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 457.189960] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 457.195939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 457.208220] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 460.890285] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 467.425791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.427042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 467.433515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.443555] Lustre: Skipped 2 previous similar messages [ 469.820628] Lustre: server umount lustre-MDT0000 complete [ 472.161675] LustreError: 19889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370202 with bad export cookie 7876357027582047669 [ 472.166434] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 472.174947] LustreError: 19889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 472.357878] Lustre: server umount lustre-MDT0001 complete [ 477.995546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.276371] LustreError: 23238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.290236] LustreError: 23238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 481.234093] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 486.586942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.840365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 486.844460] Lustre: Skipped 3 previous similar messages [ 489.683886] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 494.581698] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 498.420815] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 499.986945] Lustre: Failing over lustre-MDT0000 [ 499.995483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.003118] Lustre: Skipped 2 previous similar messages [ 500.003854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 500.011202] Lustre: Skipped 3 previous similar messages [ 500.131257] Lustre: server umount lustre-MDT0000 complete [ 505.312855] LustreError: 23965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.328819] LustreError: 23965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 515.713662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.787842] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.464957] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 521.189120] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 521.193656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 521.199220] Lustre: Skipped 1 previous similar message [ 521.207882] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 524.546626] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 531.425509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 533.428607] Lustre: server umount lustre-MDT0000 complete [ 535.625552] LustreError: 23219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370266 with bad export cookie 7876357027582048607 [ 535.629151] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.632335] LustreError: 23219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 535.820914] Lustre: server umount lustre-MDT0001 complete [ 542.040641] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:57:52 (1786370272) [ 553.364222] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 560.487570] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 587.689758] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 595.155809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.307499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 595.327722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 595.378107] Lustre: lustre-MDT0000: new disk, initializing [ 595.439682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 598.383925] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 605.801385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.880920] Lustre: 29600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 606.106495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 609.233256] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 616.336962] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 620.413595] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 624.762442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 624.949739] Lustre: 31567:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 625.025762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.034694] Lustre: Skipped 3 previous similar messages [ 626.589654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 626.624443] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 626.690453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 626.704403] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 629.099531] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 634.991188] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 639.673535] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 652.258467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 652.259229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 652.262531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 652.271591] Lustre: Skipped 3 previous similar messages [ 652.274836] Lustre: Skipped 4 previous similar messages [ 656.091644] Lustre: server umount lustre-OST0000 complete [ 664.398463] Lustre: server umount lustre-MDT0000 complete [ 666.576283] LustreError: 29590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370397 with bad export cookie 7876357027582049006 [ 666.580280] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 666.582315] LustreError: 29590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 666.860793] Lustre: server umount lustre-MDT0001 complete [ 673.271480] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 674.979521] Key type lgssc unregistered [ 675.165724] LNet: 33525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.171802] LNetError: 33525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.198341] LNet: Removed LNI 192.168.203.139@tcp [ 676.759286] Key type .llcrypt unregistered [ 676.763464] Key type ._llcrypt unregistered [ 693.481265] Key type ._llcrypt registered [ 693.483124] Key type .llcrypt registered [ 693.561916] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 694.139967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 694.179273] alg: No test for adler32 (adler32-zlib) [ 695.140570] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 695.295916] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 696.919170] Key type lgssc registered [ 697.678513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 702.303280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 702.313405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 703.525836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.720397] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 710.487902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.701290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 712.905715] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 716.872452] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 720.093443] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 723.876193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 724.056142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 727.575436] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 732.515657] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 736.361506] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 745.612252] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:01:16 (1786370476) [ 746.627301] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 747.603076] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 751.819427] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:01:22 (1786370482) [ 757.239101] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:01:27 (1786370487) [ 757.976806] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 763.473874] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 763.481750] Lustre: Skipped 2 previous similar messages [ 764.611944] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:01:35 (1786370495) [ 769.360343] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 769.364284] Lustre: Skipped 4 previous similar messages [ 778.080635] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:01:48 (1786370508) [ 779.619711] Lustre: *** cfs_fail_loc=131b, val=2*** [ 781.467458] Lustre: *** cfs_fail_loc=131b, val=2*** [ 786.024945] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:01:56 (1786370516) [ 787.276698] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 787.281556] Lustre: Skipped 9 previous similar messages [ 795.295522] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:02:05 (1786370525) [ 803.297327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 803.298877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 803.305237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 803.313225] Lustre: Skipped 1 previous similar message [ 803.322458] Lustre: Skipped 1 previous similar message [ 808.429172] Lustre: server umount lustre-OST0000 complete [ 813.024852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 813.029553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.038419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 816.684727] Lustre: server umount lustre-MDT0000 complete [ 818.656470] LustreError: 34917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.670861] LustreError: 34917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 819.083255] LustreError: 35651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370549 with bad export cookie 6691638599569633652 [ 819.085627] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 819.090136] LustreError: 35651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 819.327673] Lustre: server umount lustre-MDT0001 complete [ 827.445728] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 829.243256] Key type lgssc unregistered [ 829.440620] LNet: 42768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.444903] LNetError: 42768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.455538] LNet: Removed LNI 192.168.203.139@tcp [ 829.957314] Key type .llcrypt unregistered [ 829.960993] Key type ._llcrypt unregistered [ 840.908334] Key type ._llcrypt registered [ 840.910154] Key type .llcrypt registered [ 840.965904] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 849.907854] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 850.554715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.563424] alg: No test for adler32 (adler32-zlib) [ 851.500784] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 851.651388] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 853.287136] Key type lgssc registered [ 853.986381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 874.256668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 874.279475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.436327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 875.453827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 875.496569] Lustre: lustre-MDT0000: new disk, initializing [ 875.532023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.541251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 877.946358] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 884.341047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.426356] Lustre: 45899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 884.470166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 884.473470] Lustre: Skipped 1 previous similar message [ 884.523975] Lustre: lustre-MDT0001: new disk, initializing [ 884.584551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 884.621830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 884.640810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 887.224948] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 893.071110] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 896.559961] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 901.034361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 901.216985] Lustre: lustre-OST0000: new disk, initializing [ 901.219751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 901.224363] Lustre: 47868:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 901.292960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.098933] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 908.846882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 908.855880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 908.955676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 911.460684] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 914.388894] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 914.544100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 917.329589] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 917.477841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 919.521584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 919.522279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.526367] LustreError: Skipped 1 previous similar message [ 919.527557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 919.534687] Lustre: Skipped 1 previous similar message [ 924.642038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 924.645123] Lustre: Skipped 2 previous similar messages [ 929.762781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 929.768034] Lustre: Skipped 1 previous similar message [ 933.343219] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 933.455773] Lustre: server umount lustre-OST0000 complete [ 940.004055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 940.004958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 940.013261] Lustre: Skipped 1 previous similar message [ 940.026233] Lustre: Skipped 1 previous similar message [ 942.681883] Lustre: server umount lustre-MDT0000 complete [ 945.004505] LustreError: 46693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370675 with bad export cookie 9589094600279417995 [ 945.010190] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 945.012719] LustreError: 46693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 945.214262] Lustre: server umount lustre-MDT0001 complete [ 951.606522] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:04:41 (1786370681) [ 955.887993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.176092] LustreError: 50324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.189698] LustreError: 50324:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 956.234448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.886978] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 961.504426] LustreError: 50325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.312310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.543289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 967.206225] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 971.860378] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 975.164968] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 979.000906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 979.193437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.685874] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 986.981836] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 994.310414] LustreError: 52853:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 994.814967] Lustre: Modifying parameter general.timeout=1 in log params [ 995.838656] Lustre: Modifying parameter general.timeout=3 in log params [ 995.843132] Lustre: Skipped 1 previous similar message [ 997.992285] Lustre: Modifying parameter general.timeout=7 in log params [ 997.996264] Lustre: Skipped 3 previous similar messages [ 1002.150789] Lustre: Modifying parameter general.timeout=15 in log params [ 1002.154736] Lustre: Skipped 7 previous similar messages [ 1010.225239] Lustre: Modifying parameter general.timeout=29 in log params [ 1010.230457] Lustre: Skipped 13 previous similar messages [ 1026.440416] Lustre: Modifying parameter general.timeout=59 in log params [ 1026.442966] Lustre: Skipped 29 previous similar messages [ 1058.470251] Lustre: Modifying parameter general.timeout=118 in log params [ 1058.472944] Lustre: Skipped 58 previous similar messages [ 1069.723920] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:06:40 (1786370800) [ 1071.336215] Lustre: 59768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1073.174741] Lustre: 59965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1073.183032] Lustre: 59965:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1075.986192] Lustre: 60259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1075.997971] Lustre: 60259:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1091.106565] Lustre: 61606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1091.123890] LustreError: 61606:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1093.565500] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:07:04 (1786370824) [ 1102.113258] Lustre: server umount lustre-MDT0000 complete [ 1102.816376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1102.823197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.830733] LustreError: 50324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.949385] LustreError: 51061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370834 with bad export cookie 9589094600279418863 [ 1103.955086] LustreError: 51061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1103.956729] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1104.354670] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1106.210856] Lustre: server umount lustre-MDT0001 complete [ 1124.895548] Lustre: server umount lustre-OST0000 complete [ 1139.685187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1139.860221] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1139.877117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1139.881964] Lustre: Skipped 34 previous similar messages [ 1139.993596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.378894] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1147.045699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1147.117515] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1147.132551] Lustre: 64134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1149.464913] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1153.407090] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1156.464943] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1160.167943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1160.275584] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1163.398589] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1166.845850] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1167.666774] Lustre: 66698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1173.988449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1173.988505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.990888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1173.993149] LustreError: Skipped 1 previous similar message [ 1174.007403] Lustre: Skipped 3 previous similar messages [ 1176.684446] Lustre: server umount lustre-OST0000 complete [ 1179.105360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1184.894633] Lustre: server umount lustre-MDT0000 complete [ 1187.114723] LustreError: 64127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370917 with bad export cookie 9589094600279421439 [ 1187.120267] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1187.124792] LustreError: 64127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1187.327477] Lustre: server umount lustre-MDT0001 complete [ 1193.770597] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1195.631844] Key type lgssc unregistered [ 1195.833530] LNet: 67908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.838813] LNetError: 67908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.913225] LNet: Removed LNI 192.168.203.139@tcp [ 1197.378482] Key type .llcrypt unregistered [ 1197.381345] Key type ._llcrypt unregistered [ 1212.126563] Key type ._llcrypt registered [ 1212.127786] Key type .llcrypt registered [ 1212.184992] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1212.673706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.699522] alg: No test for adler32 (adler32-zlib) [ 1213.687143] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1213.860675] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1215.503386] Key type lgssc registered [ 1216.218826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1221.576523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1221.589737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.945085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.673644] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1230.695671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.112693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1233.795537] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1238.237718] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1241.945351] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1246.082136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1246.388916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.512915] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1253.803308] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1259.707378] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1266.146181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.148719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.159492] Lustre: Skipped 1 previous similar message [ 1266.166600] Lustre: Skipped 2 previous similar messages [ 1270.121479] Lustre: server umount lustre-MDT0000 complete [ 1271.268468] LustreError: 69300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.285274] LustreError: 69300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1272.393834] LustreError: 69280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371002 with bad export cookie 17389261826597496540 [ 1272.395806] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1272.405875] LustreError: 69280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1272.614974] Lustre: server umount lustre-MDT0001 complete [ 1285.328635] Lustre: server umount lustre-OST0000 complete [ 1299.850184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.015843] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1300.030783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1300.214447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.986845] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1308.793865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1308.855316] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1308.877255] Lustre: 74164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1309.215685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1311.903956] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1316.900787] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1320.514899] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1324.648070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1324.766871] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1325.027150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.536649] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1332.585407] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1333.613846] Lustre: 76736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1334.696735] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1340.898108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1340.899907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.902809] LustreError: Skipped 1 previous similar message [ 1340.906334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1340.910401] Lustre: Skipped 2 previous similar messages [ 1346.021340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1346.025935] Lustre: Skipped 1 previous similar message [ 1351.136459] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1351.142417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1351.154443] Lustre: Skipped 1 previous similar message [ 1351.240126] Lustre: server umount lustre-OST0000 complete [ 1355.232540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1355.237542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.245210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.249420] Lustre: Skipped 1 previous similar message [ 1359.795836] Lustre: server umount lustre-MDT0000 complete [ 1361.375880] LustreError: 75435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.386539] LustreError: 75435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1361.846695] LustreError: 74159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371092 with bad export cookie 17389261826597497387 [ 1361.851951] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1361.856128] LustreError: 74159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1362.061938] Lustre: server umount lustre-MDT0001 complete [ 1368.071387] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1369.645943] Key type lgssc unregistered [ 1369.831610] LNet: 77951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.835554] LNetError: 77951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.844140] LNet: Removed LNI 192.168.203.139@tcp [ 1370.331484] Key type .llcrypt unregistered [ 1370.333153] Key type ._llcrypt unregistered [ 1384.126683] Key type ._llcrypt registered [ 1384.129181] Key type .llcrypt registered [ 1384.186324] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1384.652666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1384.721460] alg: No test for adler32 (adler32-zlib) [ 1385.719277] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1385.861726] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1387.487155] Key type lgssc registered [ 1388.298758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1393.284634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1393.300174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.588349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.143367] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1402.248388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.674209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1404.926721] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1409.061983] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1412.158145] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1416.018526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1416.325558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.566434] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1423.689106] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1432.757641] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1433.340206] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1434.350352] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1434.354896] Lustre: Skipped 1 previous similar message [ 1436.521042] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1436.524339] Lustre: Skipped 3 previous similar messages [ 1440.795785] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1440.801342] Lustre: Skipped 7 previous similar messages [ 1448.995455] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1449.000602] Lustre: Skipped 15 previous similar messages [ 1465.121760] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1465.127498] Lustre: Skipped 27 previous similar messages [ 1490.292371] Lustre: 87419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1491.901700] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:13:42 (1786371222) [ 1492.751132] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1493.922638] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:13:44 (1786371224) [ 1497.568211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1497.571986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.578843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1498.594339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1498.594789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1498.595745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.613045] Lustre: Skipped 1 previous similar message [ 1502.354257] Lustre: server umount lustre-OST0000 complete [ 1504.736444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1504.739700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.746279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1508.833623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1508.835122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.694291] Lustre: server umount lustre-MDT0000 complete [ 1512.969125] LustreError: 79321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371243 with bad export cookie 5027074558802814319 [ 1512.974084] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1512.975393] LustreError: 79321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1513.171621] Lustre: server umount lustre-MDT0001 complete [ 1518.805893] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1520.453552] Key type lgssc unregistered [ 1520.635541] LNet: 88670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.641775] LNetError: 88670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.652877] LNet: Removed LNI 192.168.203.139@tcp [ 1521.100380] Key type .llcrypt unregistered [ 1521.102558] Key type ._llcrypt unregistered [ 1523.161819] Key type ._llcrypt registered [ 1523.163364] Key type .llcrypt registered [ 1523.231828] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1525.753253] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules [ 1526.405030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1526.430483] alg: No test for adler32 (adler32-zlib) [ 1527.430910] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1527.609412] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1529.231153] Key type lgssc registered [ 1529.922356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1530.680767] LustreError: 89869:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1541.969984] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1547.501216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1552.688229] LustreError: 89869:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1552.738585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.080080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.574287] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1562.791537] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:14:53 (1786371293) [ 1563.618745] LustreError: 90896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1563.629693] LustreError: 90896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1565.311345] LustreError: 90927:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1565.318316] LustreError: 90927:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1565.325791] LustreError: 90927:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 1565.567138] Lustre: server umount lustre-MDT0000 complete [ 1573.274305] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1574.908153] Key type lgssc unregistered [ 1575.124858] LNet: 92420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.129649] LNetError: 92420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.137680] LNet: Removed LNI 192.168.203.139@tcp [ 1575.641210] Key type .llcrypt unregistered [ 1575.643384] Key type ._llcrypt unregistered [ 1586.436863] Key type ._llcrypt registered [ 1586.438615] Key type .llcrypt registered [ 1586.503797] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1586.979369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1586.992408] alg: No test for adler32 (adler32-zlib) [ 1587.950876] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1588.078960] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1589.695170] Key type lgssc registered [ 1590.358514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1594.751781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1594.762938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.028752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.360609] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1602.825807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.194430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1605.333419] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1608.968713] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1612.069566] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1615.323952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1615.677392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1619.000463] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1623.327613] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1657.264665] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:16:27 (1786371387) [ 1658.683783] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1659.860661] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:16:30 (1786371390) [ 1663.969306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.975047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1663.977449] Lustre: Skipped 2 previous similar messages [ 1663.980224] Lustre: Skipped 1 previous similar message [ 1668.264532] Lustre: server umount lustre-MDT0000 complete [ 1669.088254] LustreError: 93624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.101231] LustreError: 93624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1670.347335] LustreError: 93602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371400 with bad export cookie 14576369087248797755 [ 1670.354683] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.512635] Lustre: server umount lustre-MDT0001 complete [ 1682.931377] Lustre: server umount lustre-OST0000 complete [ 1689.530132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1689.742862] LustreError: 97889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1689.857802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.491063] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1695.206118] LustreError: 97890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.490649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.825074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1700.424865] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1704.429347] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1707.464528] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1714.337477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1714.437275] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1714.441620] LustreError: 97877:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1714.455199] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1714.460311] LustreError: 99971:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1714.464472] LustreError: 99971:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1714.467927] LustreError: 99971:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1714.501593] Lustre: server umount lustre-OST0000 complete [ 1714.503881] LustreError: 99971:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1718.787656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1718.923695] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1718.929419] LustreError: 97877:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1718.946873] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1718.951842] LustreError: 100439:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1718.955526] LustreError: 100439:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1718.958947] LustreError: 100439:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1718.997730] Lustre: server umount lustre-OST0000 complete [ 1719.000039] LustreError: 100439:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1723.469523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1723.564603] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1723.570848] Lustre: Found index 0 for lustre-OST0000, updating log [ 1723.575408] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1723.620522] Lustre: lustre-OST0000: new disk, initializing [ 1723.624640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1723.631424] Lustre: 100972:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1723.755634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.738112] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1728.759958] Lustre: Failing over lustre-OST0000 [ 1728.826459] Lustre: server umount lustre-OST0000 complete [ 1733.600132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1733.604407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.610767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.865220] Lustre: server umount lustre-MDT0000 complete [ 1738.565661] LustreError: 99993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371469 with bad export cookie 14576369087248800499 [ 1738.567131] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1738.573028] LustreError: 99993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1738.710865] Lustre: server umount lustre-MDT0001 complete [ 1742.612428] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:17:53 (1786371473) [ 1750.094116] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1754.961350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1755.146771] LustreError: 103313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1755.199494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.149682] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1760.225270] LustreError: 103314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1761.487587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1764.075737] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1765.678416] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1769.006042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1772.394804] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1776.529221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1776.569284] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1776.623556] Lustre: lustre-OST0001: new disk, initializing [ 1776.627602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1776.631855] Lustre: 105627:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1777.747834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1777.753254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1777.795663] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1779.587592] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1784.365884] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1790.303624] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1795.554119] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.557640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1795.562985] Lustre: Skipped 3 previous similar messages [ 1795.568346] Lustre: Skipped 4 previous similar messages [ 1798.774192] Lustre: server umount lustre-MDT0001 complete [ 1800.674561] LustreError: 103314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.684545] LustreError: 103314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1804.125767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1804.196931] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1804.202415] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1804.229709] Lustre: 103301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1804.629814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1804.637490] Lustre: Skipped 3 previous similar messages [ 1806.203445] LustreError: 107260:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1806.213771] LustreError: 107260:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1806.220923] LustreError: 103305:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1806.287552] LustreError: 107278:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.139@tcp: cfg command failed: rc = -17 [ 1806.293095] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1806.293095] [ 1807.129249] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1809.898604] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1812.678154] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:19:03 (1786371543) [ 1822.049791] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1831.532714] Lustre: 109238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1844.319808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1846.271957] Lustre: 110370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1906.656297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1906.660278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.666325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1906.669682] Lustre: Skipped 1 previous similar message [ 1912.308102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1912.321281] Lustre: Skipped 7 previous similar messages [ 1912.720832] Lustre: server umount lustre-MDT0000 complete [ 1916.520418] LustreError: 103295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371647 with bad export cookie 14576369087248801493 [ 1916.528148] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1916.535107] LustreError: 103295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1917.057511] Lustre: server umount lustre-MDT0001 complete [ 1932.300791] Lustre: server umount lustre-OST0000 complete [ 1934.341271] Lustre: 92928:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786371648/real 1786371648] req@ffff9b985e5e9c00 x1873146091442816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786371664 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1934.378692] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.386260] Lustre: Skipped 3 previous similar messages [ 1935.690613] Lustre: server umount lustre-OST0001 complete [ 1940.849389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1951.645266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1961.777631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1972.257375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1988.962560] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1997.692324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.706709] Lustre: lustre-MDT0000: reset Object Index mappings [ 1998.012876] LustreError: 114323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.022968] LustreError: 114323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1998.085637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.675120] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2008.872203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.889428] Lustre: lustre-MDT0001: reset Object Index mappings [ 2013.224311] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2015.351187] Lustre: 115470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2020.908595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2023.486200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2026.310784] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2032.009400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2034.545876] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2034.550447] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2034.552525] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2037.642542] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2043.949623] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2046.770381] Lustre: 117347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2092.504908] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:23:42 (1786371822) [ 2121.702250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.704044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2121.706897] Lustre: Skipped 2 previous similar messages [ 2121.709627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2121.710403] Lustre: Skipped 1 previous similar message [ 2125.442031] Lustre: server umount lustre-MDT0000 complete [ 2126.823059] LustreError: 114320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.855041] LustreError: 114320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2128.112520] LustreError: 114303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371858 with bad export cookie 14576369087248848316 [ 2128.115328] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.131258] LustreError: 114303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2134.590497] Lustre: server umount lustre-MDT0001 complete [ 2144.040767] Lustre: server umount lustre-OST0000 complete [ 2152.854833] Lustre: server umount lustre-OST0001 complete [ 2157.977229] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 2165.291152] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2191.189623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.353582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2191.371271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2191.423900] Lustre: lustre-MDT0000: new disk, initializing [ 2191.472724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.476547] Lustre: Skipped 3 previous similar messages [ 2191.486549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2194.769456] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2199.985629] Lustre: server umount lustre-MDT0000 complete [ 2207.641472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.043336] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2208.053329] Lustre: 121618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2208.063205] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2211.668280] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2215.756889] Lustre: server umount lustre-MDT0000 complete [ 2222.354472] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:25:52 (1786371952) [ 2223.585824] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2225.050475] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:25:55 (1786371955) [ 2238.236833] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2245.262677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.980423] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2254.787267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.906275] Lustre: 123669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2254.955340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2254.960163] Lustre: Skipped 1 previous similar message [ 2255.015570] Lustre: lustre-MDT0001: new disk, initializing [ 2255.135114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2255.141379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2258.410947] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2262.276753] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2267.447794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2267.670756] Lustre: lustre-OST0000: new disk, initializing [ 2267.673525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2267.677511] Lustre: 125243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2268.960077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2268.970671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2269.160635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2272.053553] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2279.864615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2279.966733] Lustre: lustre-OST0001: new disk, initializing [ 2279.970777] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2279.977570] Lustre: 126113:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2281.510340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2281.521097] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2281.581316] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2284.129253] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2291.374345] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2294.167030] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2563.966322] Lustre: 132513:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2563.971037] Lustre: 132513:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2563.975736] Lustre: 132513:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2563.980071] Lustre: 132513:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2563.983728] Lustre: 132513:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2563.987766] Lustre: 132513:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2698.152235] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:33:48 (1786372428) [ 2711.008895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2711.012785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.017714] Lustre: Skipped 2 previous similar messages [ 2711.019852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2711.022791] Lustre: Skipped 6 previous similar messages [ 2724.319453] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2724.434460] Lustre: server umount lustre-MDT0000 complete [ 2726.218387] LustreError: 123662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786372456 with bad export cookie 14576369087248915208 [ 2726.222190] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2726.226099] LustreError: 123662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2726.880235] LustreError: 126922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.897145] LustreError: 126922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2732.474413] Lustre: server umount lustre-MDT0001 complete [ 2748.383132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2748.429394] Lustre: server umount lustre-OST0000 complete [ 2756.269484] Lustre: server umount lustre-OST0001 complete [ 2758.996635] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 2762.862472] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2777.655088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.758565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2777.779712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2777.821783] Lustre: lustre-MDT0000: new disk, initializing [ 2777.849565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2777.852026] Lustre: Skipped 5 previous similar messages [ 2777.858474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2779.550360] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2784.550396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.608360] Lustre: 135544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2784.730332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2786.484806] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2790.653702] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2792.876453] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2795.534248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2795.653176] Lustre: 137510:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2796.860893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2796.863989] Lustre: Skipped 1 previous similar message [ 2796.866138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2796.904755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2797.960342] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2801.976083] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2805.246897] Lustre: lustre-MDD0000: changelog on [ 2806.276163] Lustre: lustre-MDD0001: changelog on [ 3573.270905] Lustre: lustre-MDD0001: changelog off [ 3574.146747] Lustre: 136342:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3574.556985] Lustre: lustre-MDD0000: changelog off [ 3580.386512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3580.387349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.389534] LustreError: Skipped 1 previous similar message [ 3580.390818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3580.394356] Lustre: Skipped 6 previous similar messages [ 3580.396510] Lustre: Skipped 15 previous similar messages [ 3582.703715] Lustre: server umount lustre-OST0000 complete [ 3585.505727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.506499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3585.511140] Lustre: Skipped 1 previous similar message [ 3585.518869] Lustre: Skipped 2 previous similar messages [ 3590.824653] Lustre: server umount lustre-MDT0000 complete [ 3592.648401] LustreError: 136336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373323 with bad export cookie 14576369087249730246 [ 3592.649894] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3592.655168] LustreError: 136336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3592.809441] Lustre: server umount lustre-MDT0001 complete [ 3597.854803] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3599.403406] Key type lgssc unregistered [ 3599.577599] LNet: 146393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3599.581277] LNetError: 146393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3599.593454] LNet: Removed LNI 192.168.203.139@tcp [ 3600.031203] Key type .llcrypt unregistered [ 3600.032919] Key type ._llcrypt unregistered [ 3608.990346] Key type ._llcrypt registered [ 3608.992158] Key type .llcrypt registered [ 3609.046269] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 3617.019184] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3617.522656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3617.567363] alg: No test for adler32 (adler32-zlib) [ 3618.476566] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 3618.608967] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3620.215407] Key type lgssc registered [ 3620.816469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3638.841377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3638.855589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.998935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3640.025127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3640.071307] Lustre: lustre-MDT0000: new disk, initializing [ 3640.121026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3640.167055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3642.277859] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3648.209460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.270392] Lustre: 149566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3648.318432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3648.320919] Lustre: Skipped 1 previous similar message [ 3648.378473] Lustre: lustre-MDT0001: new disk, initializing [ 3648.438252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3648.463885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3648.472251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3650.897166] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3656.357840] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3659.246654] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3662.793942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3662.956738] Lustre: lustre-OST0000: new disk, initializing [ 3662.960598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3662.967438] Lustre: 151536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3663.026647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3663.518925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3663.532694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3663.646877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3666.367631] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3671.630149] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3674.437862] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_module obdecho/obdecho [ 3675.304531] LustreError: 152538:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3675.307645] LustreError: 152538:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3676.098704] LustreError: 152634:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3684.322069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3684.323799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.326416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3684.329484] Lustre: Skipped 2 previous similar messages [ 3684.332225] Lustre: Skipped 1 previous similar message [ 3686.367740] Lustre: server umount lustre-MDT0000 complete [ 3687.833114] LustreError: 149559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373418 with bad export cookie 11850360573186697232 [ 3687.838183] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3687.839676] LustreError: 149559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3687.999950] Lustre: server umount lustre-MDT0001 complete [ 3699.874407] Lustre: server umount lustre-OST0000 complete [ 3708.006455] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 3712.257481] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3728.142724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.268115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3728.285344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3728.339725] Lustre: lustre-MDT0000: new disk, initializing [ 3728.378927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3728.387703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3730.162395] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3735.349350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.396423] Lustre: 156341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3735.429509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3735.433113] Lustre: Skipped 1 previous similar message [ 3735.473663] Lustre: lustre-MDT0001: new disk, initializing [ 3735.529122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3735.558439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3735.565327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3737.737973] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3742.842182] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3745.380706] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3748.630704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3748.826419] Lustre: lustre-OST0000: new disk, initializing [ 3748.829711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3748.836465] Lustre: 158308:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3748.899930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3751.723577] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3754.505117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3754.511823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3754.555343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3756.665273] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3758.464149] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3758.547095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3760.265209] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3760.359810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3764.705267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3764.705422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.708399] LustreError: Skipped 1 previous similar message [ 3764.715135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.718938] Lustre: Skipped 1 previous similar message [ 3767.257517] Lustre: server umount lustre-OST0000 complete [ 3769.825742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.826772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3769.831169] Lustre: Skipped 2 previous similar messages [ 3774.758103] Lustre: server umount lustre-MDT0000 complete [ 3776.123356] LustreError: 158307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373506 with bad export cookie 11850360573186698191 [ 3776.126918] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3776.127811] LustreError: 158307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3776.259201] Lustre: server umount lustre-MDT0001 complete [ 3777.581282] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:51:48 (1786373508) [ 3778.215839] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3778.944222] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:51:49 (1786373509) [ 3782.110753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3782.346748] LustreError: 160717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3782.359336] LustreError: 160717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3782.393939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3784.192831] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3787.744610] LustreError: 160718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.959209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.906769] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3793.448212] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3796.049873] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3798.956654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3801.746207] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3804.924428] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3818.978600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3818.978762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.986793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3818.990836] Lustre: Skipped 3 previous similar messages [ 3823.900109] Lustre: server umount lustre-MDT0000 complete [ 3824.096125] LustreError: 161945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3824.102094] LustreError: 161945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3825.156311] LustreError: 162558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373555 with bad export cookie 11850360573186699059 [ 3825.159896] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3825.161251] LustreError: 162558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3825.655565] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 3825.658027] Lustre: Skipped 1 previous similar message [ 3829.215967] LustreError: 160714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3829.216239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.223076] LustreError: 160714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3829.232453] Lustre: Skipped 2 previous similar messages [ 3831.363537] Lustre: server umount lustre-MDT0001 complete [ 3833.665826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3838.158139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.243575] LustreError: 164574:0:(llog.c:1655:llog_backup()) MGC192.168.203.139@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3838.248802] Lustre: 164574:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3854.815968] LustreError: 164579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3854.821920] LustreError: 164579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3865.059681] LustreError: 164580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.068782] LustreError: 164580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3881.970635] LustreError: 164580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3881.985823] LustreError: 164580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 3898.975662] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3898.982980] Lustre: 164574:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3915.744329] LustreError: 164579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3915.753875] LustreError: 164579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 3939.816860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3939.820085] Lustre: Skipped 2 previous similar messages [ 3941.807956] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3944.935094] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3945.950315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.530989] LustreError: 147793:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b502a9bf x1873148220912512/t0(0) o250->MGC192.168.203.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3970.541619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3972.767333] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3975.883611] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3976.174492] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3976.635724] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3979.192892] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3980.047706] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3991.154175] Lustre: server umount lustre-MDT0000 complete [ 3991.521203] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.523017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3991.530232] Lustre: Skipped 2 previous similar messages [ 3991.533265] LustreError: 162547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.533277] LustreError: 162547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 40 previous similar messages [ 3992.996418] LustreError: 160697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373723 with bad export cookie 11850360573186705534 [ 3992.999225] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3993.003111] LustreError: 160697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3993.168755] Lustre: server umount lustre-MDT0001 complete [ 4005.292719] Lustre: server umount lustre-OST0000 complete [ 4016.042202] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4021.834383] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4039.683655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4039.800117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4039.824476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4039.869782] Lustre: lustre-MDT0000: new disk, initializing [ 4039.918348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.926456] Lustre: Skipped 1 previous similar message [ 4039.935757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4041.801758] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4047.466393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4047.525928] Lustre: 169651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4047.653242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4049.600594] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4054.278098] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4056.833836] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4059.632970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4059.746090] Lustre: 171621:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4062.083174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4062.132240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4062.173321] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4066.618429] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4068.421187] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4068.505893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4070.219726] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4070.317940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4072.416781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4072.417454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.421824] LustreError: Skipped 1 previous similar message [ 4072.422546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4072.427723] Lustre: Skipped 1 previous similar message [ 4072.429678] Lustre: Skipped 3 previous similar messages [ 4077.205194] Lustre: server umount lustre-OST0000 complete [ 4084.892941] Lustre: server umount lustre-MDT0000 complete [ 4086.219852] LustreError: 171620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373816 with bad export cookie 11850360573186722327 [ 4086.222825] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4086.228065] LustreError: 171620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4086.358275] Lustre: server umount lustre-MDT0001 complete [ 4090.060324] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:57:00 (1786373820) [ 4092.858635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.579179] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4097.669546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4099.238308] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4101.711322] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4103.741263] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4106.206792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4108.732431] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4111.485429] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4126.392881] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:57:37 (1786373857) [ 4128.225126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4128.226752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4128.228858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4128.233592] Lustre: Skipped 3 previous similar messages [ 4128.235900] Lustre: Skipped 2 previous similar messages [ 4133.665347] Lustre: server umount lustre-OST0000 complete [ 4141.302950] Lustre: server umount lustre-MDT0000 complete [ 4142.811158] LustreError: 174009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373873 with bad export cookie 11850360573186723209 [ 4142.814680] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4142.818538] LustreError: 174009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4142.958726] Lustre: server umount lustre-MDT0001 complete [ 4146.854740] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4147.952956] Key type lgssc unregistered [ 4148.088441] LNet: 177779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4148.091314] LNetError: 177779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4148.100419] LNet: Removed LNI 192.168.203.139@tcp [ 4148.430773] Key type .llcrypt unregistered [ 4148.432533] Key type ._llcrypt unregistered [ 4156.064328] Key type ._llcrypt registered [ 4156.065446] Key type .llcrypt registered [ 4156.099717] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4156.419526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4156.437061] alg: No test for adler32 (adler32-zlib) [ 4157.285831] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 4157.380475] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4158.959121] Key type lgssc registered [ 4159.326645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4161.847480] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4161.853838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4177.311236] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4182.431202] LustreError: 178998:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 4187.551477] LustreError: 178998:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4187.555710] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4187.557990] LustreError: MGC192.168.203.139@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4187.560865] LustreError: 178998:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4187.563846] LustreError: 178998:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 4187.566585] LustreError: 178998:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4187.585450] Lustre: server umount lustre-OST0000 complete [ 4187.586812] LustreError: 178998:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4190.354679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.503746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4192.823294] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4195.629785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.753851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4197.110401] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4199.466385] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4201.364919] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4203.667776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4203.743889] LustreError: 181311:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4203.749046] Lustre: 181311:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.139@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4203.810277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.820227] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4208.348834] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4211.172085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.172689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4211.173342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4211.177196] Lustre: Skipped 1 previous similar message [ 4215.191555] Lustre: server umount lustre-MDT0000 complete [ 4216.288723] LustreError: 179521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.295240] LustreError: 179521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4216.418234] LustreError: 179477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373946 with bad export cookie 5285987527982932237 [ 4216.421631] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4216.422423] LustreError: 179477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4216.533777] Lustre: server umount lustre-MDT0001 complete [ 4228.128298] Lustre: server umount lustre-OST0000 complete [ 4231.154433] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4234.470750] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4255.500470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4255.665748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4255.685865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4255.736252] Lustre: lustre-MDT0000: new disk, initializing [ 4255.787797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.811625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4259.886351] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4272.652723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4272.752979] Lustre: 184831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4272.802335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4272.805706] Lustre: Skipped 1 previous similar message [ 4272.910707] Lustre: lustre-MDT0001: new disk, initializing [ 4273.015444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4273.061513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4273.073312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4278.972360] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4290.386330] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4298.380562] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4308.134201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4308.806255] Lustre: lustre-OST0000: new disk, initializing [ 4308.814526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4308.839807] Lustre: 186799:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4309.173761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.849422] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4317.217349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4317.227786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4317.338124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4328.348130] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4333.283511] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4333.453238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4338.236411] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4338.535517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4342.758547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4342.762486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.793219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4342.808212] Lustre: Skipped 1 previous similar message [ 4342.823881] Lustre: Skipped 3 previous similar messages [ 4347.877126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4347.894428] Lustre: Skipped 1 previous similar message [ 4352.993190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4352.999133] Lustre: Skipped 1 previous similar message [ 4355.042079] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4355.302384] Lustre: server umount lustre-OST0000 complete [ 4360.161096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4360.165567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.172281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4363.233539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.470788] Lustre: server umount lustre-MDT0000 complete [ 4368.329739] LustreError: 184822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374098 with bad export cookie 5285987527982932937 [ 4368.344278] LustreError: 184822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4368.356267] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4368.762712] Lustre: server umount lustre-MDT0001 complete [ 4376.996880] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:01:47 (1786374107) [ 4391.901386] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4394.814252] Key type lgssc unregistered [ 4395.084791] LNet: 189759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4395.090265] LNetError: 189759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4395.103792] LNet: Removed LNI 192.168.203.139@tcp [ 4395.723853] Key type .llcrypt unregistered [ 4395.725850] Key type ._llcrypt unregistered [ 4411.075509] Key type ._llcrypt registered [ 4411.078928] Key type .llcrypt registered [ 4411.155850] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4412.715989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4412.748343] alg: No test for adler32 (adler32-zlib) [ 4414.031839] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 4414.232811] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4415.895144] Key type lgssc registered [ 4417.046328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4424.339460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4424.363530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4425.935070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4430.523237] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4436.450842] LustreError: 190973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.466212] LustreError: 190973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4438.616547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4438.847335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4442.907459] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4449.435849] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4454.004176] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4460.721745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4460.847136] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4460.849133] LustreError: 192837:0:(llog.c:1696:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4460.854327] Lustre: 192837:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.139@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4460.938620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.879110] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4473.527468] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4476.392974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4476.398099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4476.404565] Lustre: Skipped 2 previous similar messages [ 4476.420756] Lustre: Skipped 2 previous similar messages [ 4481.509971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4481.515791] Lustre: Skipped 2 previous similar messages [ 4482.043768] Lustre: server umount lustre-MDT0000 complete [ 4485.338867] LustreError: 190955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374215 with bad export cookie 2720448735101411721 [ 4485.353635] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4485.358644] LustreError: 190955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4485.771132] Lustre: server umount lustre-MDT0001 complete [ 4499.453442] Lustre: server umount lustre-OST0000 complete [ 4507.764770] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4516.900967] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4546.757774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4546.942159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4546.964017] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4547.017977] Lustre: lustre-MDT0000: new disk, initializing [ 4547.116106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4547.145569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4551.057643] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4558.192095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.323321] Lustre: 196361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4558.358785] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4558.363277] Lustre: Skipped 1 previous similar message [ 4558.365909] LustreError: 197143:0:(llog.c:1696:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4558.371671] LustreError: 197143:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4558.378231] Lustre: 197143:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4558.394178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4558.397785] Lustre: Skipped 1 previous similar message [ 4558.468160] Lustre: lustre-MDT0001: new disk, initializing [ 4558.555947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4558.587355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4558.607429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4561.804362] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4568.195395] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4571.823893] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4576.277241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4576.486311] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4576.489797] Lustre: Skipped 1 previous similar message [ 4576.493558] LustreError: 198305:0:(llog.c:1696:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4576.501274] LustreError: 198305:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4576.506468] Lustre: 198305:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4576.515284] Lustre: 198305:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4576.531860] Lustre: lustre-OST0000: new disk, initializing [ 4576.538340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4576.544791] Lustre: 198329:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4576.617477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4580.258056] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4582.412223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4582.417342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4582.468550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4586.529460] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4589.587993] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4589.738829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4593.011418] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4593.195142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4597.727988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4597.733390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4597.736091] LustreError: Skipped 1 previous similar message [ 4597.737269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4597.743052] Lustre: Skipped 1 previous similar message [ 4597.766664] Lustre: Skipped 1 previous similar message [ 4601.056913] Lustre: server umount lustre-OST0000 complete [ 4604.896445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4604.900558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.908089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4607.969997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4609.814142] LustreError: 199868:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000105880c4) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4609.854155] Lustre: server umount lustre-MDT0000 complete [ 4612.572267] LustreError: 198338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374343 with bad export cookie 2720448735101412442 [ 4612.575324] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4612.589770] LustreError: 198338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4612.830556] Lustre: server umount lustre-MDT0001 complete [ 4620.150317] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:05:50 (1786374350) [ 4633.157411] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4640.942305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4641.232528] LustreError: 201453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4641.272700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.200866] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4646.368710] LustreError: 201454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.541256] LustreError: 201453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.490942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.080507] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4659.368471] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4664.397228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4669.808285] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4675.963413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4676.066983] Lustre: lustre-OST0001: new disk, initializing [ 4676.069318] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4676.073145] Lustre: 203760:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4676.180726] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4676.189710] Lustre: Skipped 2 previous similar messages [ 4676.499172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4676.545568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4676.702243] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4680.661904] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4688.755550] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4693.356652] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4694.892643] Lustre: DEBUG MARKER: ADD OST3 [ 4698.404454] loop: module loaded [ 4698.451368] loop0: detected capacity change from 0 to 204800000 [ 4698.627709] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4698.637576] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4698.646322] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4703.126631] loop0: detected capacity change from 0 to 204800000 [ 4705.259636] Lustre: DEBUG MARKER: STOP OST3 [ 4708.060328] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4708.217288] Lustre: lustre-OST0003: new disk, initializing [ 4708.221569] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4708.229592] Lustre: 205693:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4709.683718] Lustre: Failing over lustre-OST0003 [ 4709.793893] Lustre: server umount lustre-OST0003 complete [ 4710.251403] LustreError: 202941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4713.189863] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4718.918786] Lustre: DEBUG MARKER: START OST3 again [ 4720.618805] LustreError: 202939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.639264] LustreError: 202939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4724.110962] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4729.213746] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4729.355514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4729.375680] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4729.444351] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 4746.637684] Lustre: Failing over lustre-OST0003 [ 4746.720768] Lustre: server umount lustre-OST0003 complete [ 4749.794026] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4749.805259] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4749.808900] LustreError: 202941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4749.808913] LustreError: 202941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4749.829177] Lustre: Skipped 1 previous similar message [ 4765.162026] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4765.170133] Lustre: Skipped 2 previous similar messages [ 4765.172171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4765.176573] Lustre: Skipped 5 previous similar messages [ 4767.553971] Lustre: server umount lustre-MDT0000 complete [ 4769.838364] LustreError: 201433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374500 with bad export cookie 2720448735101413317 [ 4769.855384] LustreError: 201433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4769.868363] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4774.306245] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4774.313241] Lustre: Skipped 3 previous similar messages [ 4776.143229] Lustre: server umount lustre-MDT0001 complete [ 4778.542304] Lustre: server umount lustre-OST0000 complete [ 4780.977430] Lustre: server umount lustre-OST0001 complete [ 4785.172563] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4790.294057] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4807.664944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4807.789454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4807.803733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4807.844661] Lustre: lustre-MDT0000: new disk, initializing [ 4807.881145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4807.883910] Lustre: Skipped 2 previous similar messages [ 4807.890980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4809.598082] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4814.589894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4814.651754] Lustre: 211055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4814.780025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4816.521152] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4820.673263] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4822.758507] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4825.494573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4825.617480] Lustre: lustre-OST0000: new disk, initializing [ 4825.618902] Lustre: Skipped 1 previous similar message [ 4825.621060] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4825.623980] Lustre: Skipped 2 previous similar messages [ 4825.627171] Lustre: 213021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4827.395168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4827.400370] Lustre: Skipped 1 previous similar message [ 4827.403777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4827.450199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4828.185361] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4832.566907] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4834.198761] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4834.282413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4835.926577] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4836.027051] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4837.857775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4837.858179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.860841] LustreError: Skipped 1 previous similar message [ 4837.865869] Lustre: Skipped 2 previous similar messages [ 4837.866868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4837.871974] Lustre: Skipped 1 previous similar message [ 4851.167119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4851.229251] Lustre: server umount lustre-OST0000 complete [ 4867.039118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4867.110579] Lustre: server umount lustre-MDT0000 complete [ 4868.549207] LustreError: 211046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374599 with bad export cookie 2720448735101415599 [ 4868.553241] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4868.553512] LustreError: 211046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4868.672134] Lustre: server umount lustre-MDT0001 complete [ 4869.963206] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:10:00 (1786374600) [ 4875.596599] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4879.336757] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4891.740708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4891.819971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4891.829802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4891.864379] Lustre: lustre-MDT0000: new disk, initializing [ 4891.889457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4893.247286] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4897.504781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4897.552030] Lustre: 217623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4897.653647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4899.085690] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4902.781445] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4904.707179] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4907.094031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4907.198869] Lustre: 219593:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4908.514608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4909.139857] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4912.909387] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4914.413146] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4914.494228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4915.853443] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4915.929483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4918.752967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4918.754114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4918.756051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4918.759995] Lustre: Skipped 4 previous similar messages [ 4918.762067] Lustre: Skipped 10 previous similar messages [ 4922.705295] Lustre: server umount lustre-OST0000 complete [ 4928.480271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4928.483148] LustreError: Skipped 1 previous similar message [ 4938.207115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4938.294348] Lustre: server umount lustre-MDT0000 complete [ 4939.232769] LustreError: 217632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4939.239359] LustreError: 217632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4939.988612] LustreError: 217617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374670 with bad export cookie 2720448735101416481 [ 4939.991789] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4939.992021] LustreError: 217617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4940.158755] Lustre: server umount lustre-MDT0001 complete [ 4945.615276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.989919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4945.993917] Lustre: Skipped 5 previous similar messages [ 4949.652491] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4955.672465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4959.648921] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4965.241797] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4969.737553] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4974.792670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4980.560876] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4985.964958] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5001.697271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5001.699370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5001.702909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5001.707211] Lustre: Skipped 3 previous similar messages [ 5001.715447] Lustre: Skipped 5 previous similar messages [ 5005.808854] Lustre: server umount lustre-OST0000 complete [ 5014.260984] Lustre: server umount lustre-MDT0000 complete [ 5016.074492] LustreError: 221787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374746 with bad export cookie 2720448735101417363 [ 5016.077556] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5016.084873] LustreError: 221787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5016.258483] Lustre: server umount lustre-MDT0001 complete [ 5120.992835] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 5121.995206] Key type lgssc unregistered [ 5122.129286] LNet: 225497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5122.131850] LNetError: 225497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5122.138297] LNet: Removed LNI 192.168.203.139@tcp [ 5122.453104] Key type .llcrypt unregistered [ 5122.454258] Key type ._llcrypt unregistered [ 5131.253842] Key type ._llcrypt registered [ 5131.254961] Key type .llcrypt registered [ 5131.303385] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5138.474879] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5138.809526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5138.873433] alg: No test for adler32 (adler32-zlib) [ 5139.753215] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 5139.864626] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 5141.463154] Key type lgssc registered [ 5141.911883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5155.698749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5155.706458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.802303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5156.814942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5156.853423] Lustre: lustre-MDT0000: new disk, initializing [ 5156.879643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.888109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5158.457280] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5163.043104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5163.092468] Lustre: 228820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5163.121733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5163.123962] Lustre: Skipped 1 previous similar message [ 5163.161937] Lustre: lustre-MDT0001: new disk, initializing [ 5163.200235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5163.216856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5163.221865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5164.728141] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5168.785895] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5171.047457] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5173.877788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5174.004547] Lustre: lustre-OST0000: new disk, initializing [ 5174.006903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5174.010582] Lustre: 230789:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5174.049725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5175.121351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5175.126867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5175.183093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5176.681425] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5181.078276] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5182.980951] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5183.085143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5184.945755] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5185.043252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5190.626568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.626962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5190.631872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5190.636595] Lustre: Skipped 1 previous similar message [ 5195.746542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5195.749171] Lustre: Skipped 1 previous similar message [ 5200.351117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5200.419396] Lustre: server umount lustre-OST0000 complete [ 5204.447953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5204.450905] LustreError: Skipped 1 previous similar message [ 5204.452603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.457018] Lustre: Skipped 1 previous similar message [ 5204.459783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5205.985498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.168447] Lustre: server umount lustre-MDT0000 complete [ 5209.745053] LustreError: 228813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374940 with bad export cookie 11526175795617621391 [ 5209.749765] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5209.751406] LustreError: 228813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5209.884450] Lustre: server umount lustre-MDT0001 complete [ 5217.569337] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5221.481665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5221.660461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.251042] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5226.421167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5226.557368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5228.087343] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5229.159840] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5231.542903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5233.837408] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5237.255785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.329850] Lustre: lustre-OST0001: new disk, initializing [ 5237.332788] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5237.336174] Lustre: 236036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5237.375859] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5237.379363] Lustre: Skipped 1 previous similar message [ 5239.698145] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5243.923593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5243.929480] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5243.978910] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5245.770415] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5250.667334] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5255.645705] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:16:26 (1786374986) [ 5257.184852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5257.188811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5257.194239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.197890] Lustre: Skipped 2 previous similar messages [ 5258.684130] Lustre: server umount lustre-MDT0000 complete [ 5259.233879] LustreError: 233731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5259.242318] LustreError: 233731:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5260.351709] LustreError: 233711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374990 with bad export cookie 11526175795617622259 [ 5260.354977] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5260.357709] LustreError: 233711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5260.502436] Lustre: server umount lustre-MDT0001 complete [ 5271.561871] Lustre: server umount lustre-OST0000 complete [ 5283.307193] Lustre: server umount lustre-OST0001 complete [ 5285.718274] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5289.507122] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5302.862452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5302.955528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5302.969616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5303.006747] Lustre: lustre-MDT0000: new disk, initializing [ 5303.033924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5303.043403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5304.612345] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5309.087928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5309.131304] Lustre: 239976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5309.239050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5310.764308] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5314.832862] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5316.816275] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5319.291485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5319.407386] Lustre: lustre-OST0000: new disk, initializing [ 5319.409164] Lustre: Skipped 1 previous similar message [ 5319.410911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5319.413506] Lustre: Skipped 2 previous similar messages [ 5319.415778] Lustre: 241942:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5320.889647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5320.893284] Lustre: Skipped 1 previous similar message [ 5320.896040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5320.928378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5321.544184] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5325.645573] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5327.240721] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5327.320595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5328.826393] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5328.907170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5331.427545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5331.427927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5331.431869] LustreError: Skipped 1 previous similar message [ 5331.436325] Lustre: Skipped 3 previous similar messages [ 5331.436962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5331.442433] Lustre: Skipped 1 previous similar message [ 5335.712786] Lustre: server umount lustre-OST0000 complete [ 5340.128344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5340.132109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.138049] Lustre: Skipped 1 previous similar message [ 5340.140474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5343.395918] Lustre: server umount lustre-MDT0000 complete [ 5344.930259] LustreError: 239968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375075 with bad export cookie 11526175795617624380 [ 5344.933924] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5344.935568] LustreError: 239968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5345.080332] Lustre: server umount lustre-MDT0001 complete [ 5348.303119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.494024] LustreError: 244153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5348.520367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.523214] Lustre: Skipped 2 previous similar messages [ 5350.041065] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5353.044513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5354.621860] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5357.145163] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5359.042638] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5361.386855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5363.591195] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5366.088160] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5516.261342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5516.262369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5516.263652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5516.263657] Lustre: Skipped 2 previous similar messages [ 5516.270475] Lustre: Skipped 2 previous similar messages [ 5516.282620] LustreError: Skipped 1 previous similar message [ 5518.187237] Lustre: server umount lustre-OST0000 complete [ 5526.882553] Lustre: server umount lustre-MDT0000 complete [ 5531.623697] LustreError: 246653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5531.646878] LustreError: 246653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5532.011809] LustreError: 244888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375262 with bad export cookie 11526175795617625262 [ 5532.019621] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5532.032162] LustreError: 244888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5532.408497] Lustre: server umount lustre-MDT0001 complete [ 5635.468193] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 5637.608480] Key type lgssc unregistered [ 5637.831140] LNet: 247762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5637.843264] LNetError: 247762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5637.860802] LNet: Removed LNI 192.168.203.139@tcp [ 5638.475162] Key type .llcrypt unregistered [ 5638.477599] Key type ._llcrypt unregistered [ 5664.152447] Key type ._llcrypt registered [ 5664.155734] Key type .llcrypt registered [ 5664.331521] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5678.887410] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5679.968042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5680.010215] alg: No test for adler32 (adler32-zlib) [ 5681.080363] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 5681.271520] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 5682.903288] Key type lgssc registered [ 5683.968604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5728.197514] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5738.498710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5738.512391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.733844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5739.768440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5739.817193] Lustre: lustre-MDT0000: new disk, initializing [ 5739.874466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5739.944277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5744.641581] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5754.654297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5754.773595] Lustre: 251833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5754.830610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5754.835272] Lustre: Skipped 1 previous similar message [ 5754.965882] Lustre: lustre-MDT0001: new disk, initializing [ 5755.055507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5755.084393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5755.098289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5759.745396] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5764.518668] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5771.963507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5772.177584] Lustre: lustre-OST0000: new disk, initializing [ 5772.181160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5772.186523] Lustre: 253464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5772.313544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.931123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5772.938718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5772.991401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 5779.064434] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5789.436434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5789.579793] Lustre: lustre-OST0001: new disk, initializing [ 5789.582200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5789.586295] Lustre: 254333:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5789.675486] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5795.416519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5795.425698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5795.611023] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5796.067535] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5807.683214] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5814.081198] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5826.536042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.539790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5826.543879] Lustre: Skipped 1 previous similar message [ 5826.571109] Lustre: Skipped 3 previous similar messages [ 5831.650191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5831.663935] Lustre: Skipped 3 previous similar messages [ 5836.773396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5836.779066] Lustre: Skipped 3 previous similar messages [ 5839.347760] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5839.572573] Lustre: server umount lustre-MDT0000 complete [ 5843.708658] LustreError: 251825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375574 with bad export cookie 12821156332877288494 [ 5843.724201] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5844.020494] Lustre: server umount lustre-MDT0001 complete [ 5857.878663] Lustre: server umount lustre-OST0000 complete [ 5870.767911] Lustre: server umount lustre-OST0001 complete [ 5884.883852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5894.141584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5902.380341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5911.292664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5931.944865] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5941.944189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5941.962404] Lustre: lustre-MDT0000: reset Object Index mappings [ 5942.239639] LustreError: 259346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5942.258921] LustreError: 259346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5942.311282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5946.516858] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5947.362761] LustreError: 259347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.481233] LustreError: 259346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5955.390413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5955.401579] Lustre: lustre-MDT0001: reset Object Index mappings [ 5955.688904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5959.918419] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5963.345690] Lustre: 260487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5970.135885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5977.069966] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5980.649267] LustreError: 260842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5984.564524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5984.757276] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5984.766575] Lustre: Skipped 1 previous similar message [ 5986.796607] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5986.802469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 5990.241911] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5996.690196] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6000.307387] Lustre: 262361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6006.734921] Lustre: server umount lustre-MDT0000 complete [ 6006.752665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6006.762098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.776739] Lustre: Skipped 2 previous similar messages [ 6006.785587] LustreError: 259347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6010.144491] LustreError: 259327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375740 with bad export cookie 12821156332877295284 [ 6010.164795] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6010.436206] Lustre: server umount lustre-MDT0001 complete [ 6024.228280] Lustre: server umount lustre-OST0000 complete [ 6027.552154] Lustre: 249342:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786375742/real 1786375742] req@ffff9b9848101c00 x1873150383090560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786375758 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6027.577773] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6027.586690] Lustre: Skipped 3 previous similar messages [ 6027.837786] Lustre: server umount lustre-OST0001 complete [ 6045.276870] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:29:35 (1786375775) [ 6060.500963] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6069.557039] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6100.289681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.414257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6100.433232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6100.486951] Lustre: lustre-MDT0000: new disk, initializing [ 6100.540906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.559414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6104.506371] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6112.645785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.804196] Lustre: 266613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6112.843594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6112.845929] Lustre: Skipped 1 previous similar message [ 6112.894446] Lustre: lustre-MDT0001: new disk, initializing [ 6113.038983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6113.050831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6116.562677] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6124.742581] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6130.875415] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6137.259256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6137.468768] Lustre: lustre-OST0000: new disk, initializing [ 6137.473114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6137.478950] Lustre: 268579:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6138.883165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6138.904460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6139.037893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6142.321188] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6149.901063] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6153.815994] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6153.973954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6157.951293] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6158.155072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6159.840506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6159.844537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6159.851485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6164.456382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6164.461466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6164.463977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.486833] Lustre: Skipped 1 previous similar message [ 6165.773871] Lustre: server umount lustre-OST0000 complete [ 6169.576818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6174.689107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6174.698310] Lustre: Skipped 3 previous similar messages [ 6183.391244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6183.544440] Lustre: server umount lustre-MDT0000 complete [ 6184.930639] LustreError: 267488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.960390] LustreError: 267488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6186.812460] LustreError: 266603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375917 with bad export cookie 12821156332877297377 [ 6186.814707] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6186.818048] LustreError: 266603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6187.139387] Lustre: server umount lustre-MDT0001 complete [ 6203.903365] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6213.060767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6213.351334] LustreError: 271516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6213.364289] LustreError: 271516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6213.417756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6213.423599] Lustre: Skipped 2 previous similar messages [ 6217.128703] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6224.037543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.355453] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6230.646495] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6235.568918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6240.717344] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6248.360095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6248.508432] Lustre: lustre-OST0001: new disk, initializing [ 6248.512729] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6248.517238] Lustre: 273822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6249.876840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6249.887994] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6249.988268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6253.470737] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6261.059758] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6263.970944] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6268.411684] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6268.438166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6275.438142] Lustre: server umount lustre-MDT0000 complete [ 6275.560412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6275.565263] LustreError: 273021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6275.572648] Lustre: Skipped 3 previous similar messages [ 6275.573335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6275.582351] LustreError: 273021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6278.185147] LustreError: 271496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376008 with bad export cookie 12821156332877298259 [ 6278.190372] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6278.193838] LustreError: 271496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6278.446651] Lustre: server umount lustre-MDT0001 complete [ 6291.896633] Lustre: server umount lustre-OST0000 complete [ 6303.828586] Lustre: server umount lustre-OST0001 complete [ 6317.962921] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6326.087688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.796723] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6335.779346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.970930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6335.977119] Lustre: Skipped 2 previous similar messages [ 6339.294905] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6341.240762] Lustre: 278015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6345.770073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6345.999163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6346.003679] Lustre: Skipped 5 previous similar messages [ 6351.140700] LustreError: 278367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6351.171795] LustreError: 278367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6351.191847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6351.510545] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6358.697557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6360.952752] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6363.350710] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6369.489472] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6372.306642] Lustre: 279889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6376.602433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6376.661670] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6384.071566] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:35:14 (1786376114) [ 6397.688592] Lustre: server umount lustre-MDT0000 complete [ 6400.506646] LustreError: 278393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376131 with bad export cookie 12821156332877304587 [ 6400.511153] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6400.536930] LustreError: 278393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6402.017528] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.020638] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6402.033755] Lustre: Skipped 4 previous similar messages [ 6406.867585] Lustre: server umount lustre-MDT0001 complete [ 6415.624803] Lustre: server umount lustre-OST0000 complete [ 6424.351921] Lustre: server umount lustre-OST0001 complete [ 6429.601406] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6438.314588] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6473.081291] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6481.154906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.343900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6481.377100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6481.441601] Lustre: lustre-MDT0000: new disk, initializing [ 6481.532693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6484.650192] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6492.542102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6492.666129] Lustre: 284127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6492.898525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6496.268212] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6504.753870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6504.992431] Lustre: 285757:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6506.925830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6509.778656] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6517.797367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6517.915057] Lustre: 286628:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6520.071410] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6522.317388] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6529.552837] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6532.025581] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6532.031019] Lustre: Skipped 1 previous similar message [ 6542.660559] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:37:52 (1786376272) [ 6544.353950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6544.360745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6544.379183] Lustre: Skipped 1 previous similar message [ 6544.382745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6544.386804] Lustre: Skipped 3 previous similar messages [ 6550.335746] Lustre: server umount lustre-MDT0000 complete [ 6551.012189] LustreError: 284135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6551.029558] LustreError: 284135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6552.628333] LustreError: 284118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376283 with bad export cookie 12821156332877312455 [ 6552.632967] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6552.637901] LustreError: 284118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6552.873099] Lustre: server umount lustre-MDT0001 complete [ 6565.448108] Lustre: server umount lustre-OST0000 complete [ 6578.461097] Lustre: server umount lustre-OST0001 complete [ 6584.521292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6587.727204] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6593.436865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6596.462755] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6601.309351] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6605.384961] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6610.264537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6610.553065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6610.568740] Lustre: Skipped 7 previous similar messages [ 6615.170935] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6620.720756] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6642.227156] loop1: detected capacity change from 0 to 204800000 [ 6642.398130] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6642.404850] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6642.417580] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6645.748050] loop1: detected capacity change from 0 to 204800000 [ 6650.088026] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6650.156183] LustreError: 288862:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs30436: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6650.165252] LustreError: 288862:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs30436 [ 6650.169415] LustreError: 288862:0:(mgs_handler.c:517:mgs_target_reg()) fs30436-MDT0001: Error checking target index: rc = -13 [ 6650.175990] LustreError: fs30436-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6650.183092] LustreError: 292616:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6650.187547] LustreError: 292616:0:(tgt_mount.c:2129:server_put_super()) no obd fs30436-MDT0001 [ 6650.190797] LustreError: 292616:0:(tgt_mount.c:116:server_deregister_mount()) fs30436-MDT0001 not registered [ 6650.212835] Lustre: server umount fs30436-MDT0001 complete [ 6650.214869] LustreError: 292616:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6656.355476] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.410859] Lustre: Setting parameter fs30436-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs30436-MDT0001 [ 6656.440981] Lustre: srv-fs30436-MDT0001: No data found on store. Initialize space. [ 6656.443959] Lustre: Skipped 4 previous similar messages [ 6656.501127] Lustre: fs30436-MDT0001: new disk, initializing [ 6656.503718] Lustre: Skipped 3 previous similar messages [ 6656.587096] Lustre: srv-fs30436-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6656.598383] Lustre: 293082:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30436-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6656.610586] LustreError: 293082:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30436-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6659.445491] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6664.490226] Lustre: server umount fs30436-MDT0001 complete [ 6674.523495] loop1: detected capacity change from 0 to 204800000 [ 6678.352728] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6678.547766] Lustre: 294647:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30436-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6678.557799] LustreError: 294647:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30436-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6681.371030] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6684.574441] Lustre: server umount fs30436-MDT0001 complete [ 6691.816708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.827481] Lustre: Skipped 4 previous similar messages [ 6691.834284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6691.841233] Lustre: Skipped 6 previous similar messages [ 6697.369914] Lustre: server umount lustre-MDT0000 complete [ 6702.060738] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 6703.546749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.619904] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6706.722136] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6709.227801] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6712.173511] loop1: detected capacity change from 0 to 204800000 [ 6716.238134] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.498118] Lustre: 297093:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30436-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6716.504044] LustreError: 297093:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30436-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6719.636490] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6722.749585] Lustre: server umount fs30436-MDT0001 complete [ 6739.940253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6743.761706] Lustre: server umount lustre-OST0000 complete [ 6752.713374] Lustre: server umount lustre-MDT0000 complete [ 6755.198965] LustreError: 295140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376485 with bad export cookie 12821156332877315136 [ 6755.204786] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6755.206831] LustreError: 295140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6755.624892] Lustre: server umount lustre-MDT0001 complete [ 6763.551453] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 6765.548046] Key type lgssc unregistered [ 6765.775449] LNet: 299437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6765.780779] LNetError: 299437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6766.821934] LNet: Removed LNI 192.168.203.139@tcp [ 6767.433143] Key type .llcrypt unregistered [ 6767.434653] Key type ._llcrypt unregistered [ 6780.772477] Key type ._llcrypt registered [ 6780.774616] Key type .llcrypt registered [ 6780.859504] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6781.496112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6781.592055] alg: No test for adler32 (adler32-zlib) [ 6782.587133] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 6782.780027] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 6784.447448] Key type lgssc registered [ 6785.320995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6790.687689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6790.700486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6791.997874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6795.106051] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6800.638032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.914798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6803.958596] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6809.194692] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6813.396833] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6817.789842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6818.037183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6822.156971] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6826.214528] LustreError: 303138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6826.229839] LustreError: 303138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6827.151863] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6831.594023] LustreError: 302517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6836.708780] LustreError: 302517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6836.726739] LustreError: 302517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6838.792459] LustreError: 303189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6843.892509] LustreError: 302514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6843.900716] LustreError: 302514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6850.458672] loop1: detected capacity change from 0 to 204800000 [ 6850.641953] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6850.653103] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6850.668168] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6852.071157] LustreError: 303189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6852.081057] LustreError: 303189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6854.467745] loop1: detected capacity change from 0 to 204800000 [ 6858.036734] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6858.143426] Lustre: fs18539-OST0000: new disk, initializing [ 6858.146723] Lustre: srv-fs18539-OST0000: No data found on store. Initialize space. [ 6858.151731] Lustre: 304441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6858.206657] Lustre: fs18539-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6862.207841] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6868.317905] Lustre: server umount fs18539-OST0000 complete [ 6869.500758] LustreError: 302517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6869.510414] LustreError: 302517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6876.105235] loop1: detected capacity change from 0 to 204800000 [ 6879.768793] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6879.910867] Lustre: fs18539-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6884.075614] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6890.015951] loop2: detected capacity change from 0 to 204800000 [ 6890.215587] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6890.224567] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6890.233876] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6893.771265] loop2: detected capacity change from 0 to 204800000 [ 6897.128813] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6897.169159] LustreError: fs18539-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6897.174702] LustreError: fs18539-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6897.182206] LustreError: 307477:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6897.187273] LustreError: 307477:0:(tgt_mount.c:2129:server_put_super()) no obd fs18539-OST0001 [ 6897.191396] LustreError: 307477:0:(tgt_mount.c:116:server_deregister_mount()) fs18539-OST0001 not registered [ 6897.222468] Lustre: server umount fs18539-OST0001 complete [ 6897.225240] LustreError: 307477:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6903.271365] LustreError: 302517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6903.283144] LustreError: 302517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 6903.388391] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6903.468754] Lustre: fs18539-OST0001: new disk, initializing [ 6903.472767] Lustre: srv-fs18539-OST0001: No data found on store. Initialize space. [ 6903.478418] Lustre: 308029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6903.517798] Lustre: fs18539-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6907.012807] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6912.195469] Lustre: server umount fs18539-OST0000 complete [ 6913.511040] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 6917.638295] Lustre: server umount fs18539-OST0001 complete [ 6918.633113] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 6925.364918] loop1: detected capacity change from 0 to 204800000 [ 6929.247882] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6929.395387] Lustre: fs18539-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6933.953842] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6938.799970] loop2: detected capacity change from 0 to 204800000 [ 6942.608607] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6946.175518] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6951.591894] Lustre: server umount fs18539-OST0000 complete [ 6957.541560] Lustre: server umount fs18539-OST0001 complete [ 6969.826046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6969.827787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.832436] LustreError: 302516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6969.832451] LustreError: 302516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 6969.833331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6969.833867] LustreError: Skipped 1 previous similar message [ 6969.844773] Lustre: Skipped 1 previous similar message [ 6971.459978] Lustre: server umount lustre-OST0000 complete [ 6974.945855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.951345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6974.952193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6974.952198] Lustre: Skipped 1 previous similar message [ 6974.954294] Lustre: Skipped 1 previous similar message [ 6980.068138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6980.078754] Lustre: Skipped 2 previous similar messages [ 6980.365025] Lustre: server umount lustre-MDT0000 complete [ 6982.712585] LustreError: 300669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376713 with bad export cookie 8718638790923754152 [ 6982.717458] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6982.719103] LustreError: 300669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6982.918572] Lustre: server umount lustre-MDT0001 complete [ 6989.155038] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 6990.671215] Key type lgssc unregistered [ 6990.833650] LNet: 313950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6990.838260] LNetError: 313950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6990.850539] LNet: Removed LNI 192.168.203.139@tcp [ 6991.384227] Key type .llcrypt unregistered [ 6991.386222] Key type ._llcrypt unregistered [ 7011.618883] Key type ._llcrypt registered [ 7011.621469] Key type .llcrypt registered [ 7011.808981] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7023.065830] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7023.902405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7023.917425] alg: No test for adler32 (adler32-zlib) [ 7024.861510] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 7025.030774] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7026.679202] Key type lgssc registered [ 7027.565590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7052.216692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7052.232732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.373330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7053.392384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7053.444405] Lustre: lustre-MDT0000: new disk, initializing [ 7053.504571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7053.521903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7056.152969] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7063.164310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.272754] Lustre: 317787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7063.318952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7063.323531] Lustre: Skipped 1 previous similar message [ 7063.379384] Lustre: lustre-MDT0001: new disk, initializing [ 7063.446551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7063.466506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7063.486196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7065.963170] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7071.809712] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7075.142264] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7079.536972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7079.803325] Lustre: lustre-OST0000: new disk, initializing [ 7079.806677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7079.814237] Lustre: 319751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7079.884207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7084.074811] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7085.125851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7085.138823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7085.251267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7090.907929] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7092.290612] Lustre: Mounted lustre-client - version 2.17.56_50_g6d2b241 [ 7093.000701] Lustre: Unmounted lustre-client [ 7100.897532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7100.909130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.911343] LustreError: Skipped 1 previous similar message [ 7100.912752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7100.939945] Lustre: Skipped 1 previous similar message [ 7102.437107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7102.443842] Lustre: Skipped 2 previous similar messages [ 7105.226695] Lustre: server umount lustre-OST0000 complete [ 7109.600629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7109.620671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7109.629065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7112.673871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.674720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7112.688873] Lustre: Skipped 1 previous similar message [ 7113.797461] Lustre: server umount lustre-MDT0000 complete [ 7116.157368] LustreError: 317778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376846 with bad export cookie 17783093107937086859 [ 7116.160888] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7116.169266] LustreError: 317778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7116.369635] Lustre: server umount lustre-MDT0001 complete [ 7123.199869] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7125.153855] Key type lgssc unregistered [ 7144.799905] LNet: 321917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7144.804762] LNetError: 321917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7222.629230] LNet: Removed LNI 192.168.203.139@tcp [ 7223.255261] Key type .llcrypt unregistered [ 7223.257982] Key type ._llcrypt unregistered [ 7235.636345] Key type ._llcrypt registered [ 7235.638633] Key type .llcrypt registered [ 7235.718127] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7245.825651] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7246.413878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7246.461064] alg: No test for adler32 (adler32-zlib) [ 7247.441422] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 7247.627454] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7249.279169] Key type lgssc registered [ 7250.050778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7270.772029] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:50:01 (1786377001) [ 7275.271438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7275.281534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.408328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7276.427363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7276.481496] Lustre: lustre-MDT0000: new disk, initializing [ 7276.539100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7276.554635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7279.248656] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7286.028855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7286.113314] Lustre: 325117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7286.153154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7286.158103] Lustre: Skipped 1 previous similar message [ 7286.201661] Lustre: lustre-MDT0001: new disk, initializing [ 7286.264468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7286.283965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7286.288349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7288.938896] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7295.023971] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7298.341563] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7302.386773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7302.556449] Lustre: lustre-OST0000: new disk, initializing [ 7302.558914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7302.563661] Lustre: 327084:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7302.637851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7306.169188] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7308.315597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7308.322505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7308.400040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7312.163466] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7323.622269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7323.626725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7323.636232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7323.641384] Lustre: Skipped 1 previous similar message [ 7327.371743] Lustre: server umount lustre-OST0000 complete [ 7332.321110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7332.327975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.341714] Lustre: Skipped 1 previous similar message [ 7332.345765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7333.856732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7333.861351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.226943] Lustre: server umount lustre-MDT0000 complete [ 7338.785594] LustreError: 325111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377069 with bad export cookie 7910220597293276503 [ 7338.789960] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.795675] LustreError: 325111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7339.030854] Lustre: server umount lustre-MDT0001 complete [ 7345.868454] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7347.611789] Key type lgssc unregistered [ 7347.812217] LNet: 328991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7347.817422] LNetError: 328991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7347.830890] LNet: Removed LNI 192.168.203.139@tcp [ 7348.363356] Key type .llcrypt unregistered [ 7348.366058] Key type ._llcrypt unregistered [ 7361.214823] Key type ._llcrypt registered [ 7361.219239] Key type .llcrypt registered [ 7361.291708] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7362.061458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7362.077974] alg: No test for adler32 (adler32-zlib) [ 7363.035176] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 7363.190372] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7364.800984] Key type lgssc registered [ 7365.641775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7371.120954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7371.132434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7372.458839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7375.776918] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7381.985578] LustreError: 330244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7381.995412] LustreError: 330244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7382.137396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7382.368627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7385.436885] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7390.106516] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7393.341449] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7397.451024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7397.636601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7401.218831] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7406.059112] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7423.795604] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7424.983395] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7428.107579] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 7428.576922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7428.584073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.684129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7432.686732] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.688884] Lustre: Skipped 1 previous similar message [ 7432.714612] Lustre: Skipped 1 previous similar message [ 7433.885662] Lustre: server umount lustre-MDT0000 complete [ 7436.650144] LustreError: 330223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377167 with bad export cookie 12879326145821149358 [ 7436.654171] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7436.660808] LustreError: 330223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7437.791963] LustreError: 330238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.793351] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7437.823094] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7437.826313] Lustre: Skipped 3 previous similar messages [ 7441.825087] LustreError: 332538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7441.825919] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7441.840476] LustreError: 332538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7441.856919] Lustre: Skipped 1 previous similar message [ 7442.915974] Lustre: server umount lustre-MDT0001 complete [ 7451.876699] Lustre: server umount lustre-OST0000 complete [ 7457.817779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7460.923625] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7462.552261] Lustre: 334441:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.139@tcp [ 7463.664577] Lustre: server umount MGS complete [ 7468.806386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.062431] LustreError: 335012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7469.123030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7471.813738] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7474.145985] LustreError: 335013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7478.282205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7478.541582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7481.685907] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7487.319325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7487.538936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7491.261588] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7495.656412] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7496.601823] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7500.087191] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7500.994919] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7503.841223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7503.844752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7503.851709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7504.866283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7509.670738] Lustre: server umount lustre-OST0000 complete [ 7514.592353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7514.596357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.601804] Lustre: Skipped 1 previous similar message [ 7514.604915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7514.608317] Lustre: Skipped 2 previous similar messages [ 7517.723589] Lustre: server umount lustre-MDT0000 complete [ 7519.826608] LustreError: 337492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377250 with bad export cookie 12879326145821150198 [ 7519.830665] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7519.835996] LustreError: 337492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7520.009952] Lustre: server umount lustre-MDT0001 complete [ 7525.681402] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7527.340502] Key type lgssc unregistered [ 7527.558180] LNet: 338446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7527.566481] LNetError: 338446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7527.579930] LNet: Removed LNI 192.168.203.139@tcp [ 7528.150161] Key type .llcrypt unregistered [ 7528.152560] Key type ._llcrypt unregistered [ 7539.911772] Key type ._llcrypt registered [ 7539.914732] Key type .llcrypt registered [ 7539.999497] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7550.201748] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7550.723829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7550.740061] alg: No test for adler32 (adler32-zlib) [ 7551.706248] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 7551.897888] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7553.551279] Key type lgssc registered [ 7554.392233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7577.428519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7577.440391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7578.611163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7578.632268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7578.687293] Lustre: lustre-MDT0000: new disk, initializing [ 7578.731535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.745039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7581.882852] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7589.459970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.546146] Lustre: 341585:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7589.585990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7589.588662] Lustre: Skipped 1 previous similar message [ 7589.646691] Lustre: lustre-MDT0001: new disk, initializing [ 7589.714206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7589.739464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7589.754558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7592.231429] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7598.065966] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7601.460321] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7605.272182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7605.422068] Lustre: lustre-OST0000: new disk, initializing [ 7605.427286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7605.435693] Lustre: 343556:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7605.502133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7609.251794] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7611.400317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7611.405560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7611.455341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7615.655924] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7618.344556] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7618.459690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7621.555729] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7621.798239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7626.722353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7626.723959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7626.730173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7626.734928] Lustre: Skipped 1 previous similar message [ 7626.749445] Lustre: Skipped 1 previous similar message [ 7629.355361] Lustre: server umount lustre-OST0000 complete [ 7631.842705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.844471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7631.857294] Lustre: Skipped 1 previous similar message [ 7636.965202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7636.967449] Lustre: Skipped 2 previous similar messages [ 7637.959694] Lustre: server umount lustre-MDT0000 complete [ 7640.992839] LustreError: 342382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377371 with bad export cookie 7866385086439417546 [ 7640.994479] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7640.998943] LustreError: 342382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7641.196919] Lustre: server umount lustre-MDT0001 complete [ 7648.124320] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:56:18 (1786377378) [ 7661.863286] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7668.836331] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7692.651474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7692.852107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7692.902448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7692.977267] Lustre: lustre-MDT0000: new disk, initializing [ 7693.044465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7693.063031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7695.626221] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7702.657690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7702.770802] Lustre: 348309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7702.820301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7702.823789] Lustre: Skipped 1 previous similar message [ 7702.879726] Lustre: lustre-MDT0001: new disk, initializing [ 7702.948305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7702.985285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7702.998316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7706.136277] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7712.925166] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7716.333320] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7720.914723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7721.089337] Lustre: lustre-OST0000: new disk, initializing [ 7721.092915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7721.099635] Lustre: 350274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7721.163470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7725.648387] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7730.733171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7730.742930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7730.841357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7732.711883] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7756.770078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7756.772197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7756.775581] LustreError: Skipped 1 previous similar message [ 7756.777596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7756.802022] Lustre: Skipped 1 previous similar message [ 7758.444958] Lustre: server umount lustre-OST0000 complete [ 7761.888833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.899290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.903282] Lustre: Skipped 2 previous similar messages [ 7767.177950] Lustre: server umount lustre-MDT0000 complete [ 7769.232262] LustreError: 351144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377499 with bad export cookie 7866385086439418428 [ 7769.235493] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7769.241373] LustreError: 351144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7769.428975] Lustre: server umount lustre-MDT0001 complete [ 7775.112221] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7776.562645] Key type lgssc unregistered [ 7776.729805] LNet: 352428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7776.735548] LNetError: 352428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7776.743482] LNet: Removed LNI 192.168.203.139@tcp [ 7777.210902] Key type .llcrypt unregistered [ 7777.213600] Key type ._llcrypt unregistered [ 7830.333417] Key type ._llcrypt registered [ 7830.335172] Key type .llcrypt registered [ 7830.421947] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7830.688781] Key type .llcrypt unregistered [ 7830.690993] Key type ._llcrypt unregistered [ 7876.863983] Key type ._llcrypt registered [ 7876.866240] Key type .llcrypt registered [ 7876.973623] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7877.204304] Key type .llcrypt unregistered [ 7877.206136] Key type ._llcrypt unregistered [ 7893.867751] Key type ._llcrypt registered [ 7893.869810] Key type .llcrypt registered [ 7893.935506] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7894.148289] Key type .llcrypt unregistered [ 7894.152215] Key type ._llcrypt unregistered [ 7918.566509] Key type ._llcrypt registered [ 7918.574253] Key type .llcrypt registered [ 7918.735628] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7919.023487] Key type .llcrypt unregistered [ 7919.029081] Key type ._llcrypt unregistered [ 8021.773457] Key type ._llcrypt registered [ 8021.775642] Key type .llcrypt registered [ 8022.021988] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8022.556678] Key type .llcrypt unregistered [ 8022.558798] Key type ._llcrypt unregistered [ 8054.696943] Key type ._llcrypt registered [ 8054.699558] Key type .llcrypt registered [ 8054.793448] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8055.235253] Key type .llcrypt unregistered [ 8055.247878] Key type ._llcrypt unregistered [ 8085.019202] Key type ._llcrypt registered [ 8085.021343] Key type .llcrypt registered [ 8085.101412] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8085.570396] Key type .llcrypt unregistered [ 8085.576609] Key type ._llcrypt unregistered [ 8116.896212] Key type ._llcrypt registered [ 8116.898125] Key type .llcrypt registered [ 8116.982195] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8117.289740] Key type .llcrypt unregistered [ 8117.294758] Key type ._llcrypt unregistered [ 8148.123541] Key type ._llcrypt registered [ 8148.126155] Key type .llcrypt registered [ 8148.231701] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8148.567889] Key type .llcrypt unregistered [ 8148.574584] Key type ._llcrypt unregistered [ 8178.759569] Key type ._llcrypt registered [ 8178.762500] Key type .llcrypt registered [ 8178.865710] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8179.429408] Key type .llcrypt unregistered [ 8179.431784] Key type ._llcrypt unregistered [ 8246.182343] Key type ._llcrypt registered [ 8246.184598] Key type .llcrypt registered [ 8246.258738] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8246.625519] Key type .llcrypt unregistered [ 8246.628954] Key type ._llcrypt unregistered [ 8276.840881] Key type ._llcrypt registered [ 8276.847509] Key type .llcrypt registered [ 8276.933219] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8277.150343] Key type .llcrypt unregistered [ 8277.154437] Key type ._llcrypt unregistered [ 8310.275487] Key type ._llcrypt registered [ 8310.278571] Key type .llcrypt registered [ 8310.395491] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8310.759298] Key type .llcrypt unregistered [ 8310.761730] Key type ._llcrypt unregistered [ 8370.220366] Key type ._llcrypt registered [ 8370.222730] Key type .llcrypt registered [ 8370.296557] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8385.627212] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8387.928526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8388.031129] alg: No test for adler32 (adler32-zlib) [ 8389.107321] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 8389.358354] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8391.097988] Key type lgssc registered [ 8392.204978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8432.755597] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8432.783713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8434.002383] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8434.022989] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8434.153191] Lustre: tst250fs-MDT0000: new disk, initializing [ 8434.203851] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8434.216220] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8438.615846] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8449.699762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.873546] Lustre: 372596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8449.911215] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8449.914457] Lustre: Skipped 1 previous similar message [ 8449.967294] Lustre: tst250fs-MDT0001: new disk, initializing [ 8450.054783] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8450.105898] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8450.127878] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8455.175457] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8459.802569] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8467.262130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8467.505932] Lustre: tst250fs-OST0000: new disk, initializing [ 8467.513089] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8467.525249] Lustre: tst250fs-OST0000: Not available for connect from 0@lo (not set up) [ 8467.527112] Lustre: 374232:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8467.658722] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8473.143866] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8473.158951] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8473.194935] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 8474.118209] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8484.664431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8484.873509] Lustre: tst250fs-OST0001: new disk, initializing [ 8484.876889] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8484.883438] Lustre: 375101:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8485.014390] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8492.443521] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8494.768521] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8494.779394] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8494.891345] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8499.177651] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8501.651662] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [ 8501.658740] Lustre: Skipped 2 previous similar messages [ 8520.808455] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8520.824201] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8520.867642] Lustre: Skipped 3 previous similar messages [ 8520.905345] Lustre: Skipped 3 previous similar messages [ 8525.805587] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8525.810411] Lustre: Skipped 3 previous similar messages [ 8526.592192] Lustre: server umount tst250fs-MDT0000 complete [ 8530.328658] LustreError: 372587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786378260 with bad export cookie 9420911971926754361 [ 8530.339809] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8530.416563] LustreError: 372587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8530.994652] Lustre: tst250fs-MDT0001-lwp-OST0000: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8531.079647] Lustre: server umount tst250fs-MDT0001 complete [ 8536.851832] Lustre: server umount tst250fs-OST0000 complete [ 8540.887730] Lustre: server umount tst250fs-OST0001 complete [ 8558.946560] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8567.107423] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8600.828272] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:12:10 (1786378330) [ 8602.306124] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8620.259652] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8630.690584] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8664.049310] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8407 sec ========== 12:13:14 (1786378394) [ 8666.232358] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:13:15 (1786378395) === [ 8669.482506] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:13:19 (1786378399) === [ 8708.056746] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8711.509831] Key type lgssc unregistered [ 8711.758572] LNet: 384230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8711.777387] LNetError: 384230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8711.810562] LNet: Removed LNI 192.168.203.139@tcp [ 8712.894327] Key type .llcrypt unregistered [ 8712.897972] Key type ._llcrypt unregistered