[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496884585 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003207] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007604] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009025] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.022188] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026604] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] *** VALIDATE tmpfs *** [ 0.030493] *** VALIDATE proc *** [ 0.032200] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035037] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042565] debug: unmapping init [mem 0xffffffffb6a59000-0xffffffffb6a60fff] [ 0.044175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045654] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047008] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049008] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.054186] rcu: Hierarchical SRCU implementation. [ 0.056298] smp: Bringing up secondary CPUs ... [ 0.057604] x86: Booting SMP configuration: [ 0.058031] .... node #0, CPUs: #1 #2 #3 [ 0.062025] smp: Brought up 1 node, 4 CPUs [ 0.064017] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.229118] node 0 deferred pages initialised in 160ms [ 0.232013] devtmpfs: initialized [ 0.233307] x86/mm: Memory block size: 128MB [ 0.235866] gcov: version magic: 0x41383552 [ 0.237200] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.238085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.239256] pinctrl core: initialized pinctrl subsystem [ 0.240209] [ 0.240802] ************************************************************* [ 0.241020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.242018] ** ** [ 0.243017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.244018] ** ** [ 0.245017] ** This means that this kernel is built to expose internal ** [ 0.246018] ** IOMMU data structures, which may compromise security on ** [ 0.247016] ** your system. ** [ 0.248016] ** ** [ 0.249019] ** If you see this message and you are not debugging the ** [ 0.250019] ** kernel, report this immediately to your vendor! ** [ 0.251016] ** ** [ 0.252017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253033] ************************************************************* [ 0.254681] NET: Registered protocol family 16 [ 0.255462] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.256066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.257075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.258487] cpuidle: using governor menu [ 0.260770] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.262687] PCI: Using configuration type 1 for base access [ 0.265159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.275062] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.276043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.278037] cryptd: max_cpu_qlen set to 1000 [ 0.280249] ACPI: Added _OSI(Module Device) [ 0.281021] ACPI: Added _OSI(Processor Device) [ 0.282018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.283018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.287313] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.291221] ACPI: Interpreter enabled [ 0.292069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.293023] ACPI: Using IOAPIC for interrupt routing [ 0.294113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.295401] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.304927] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.305052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.306026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.307083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.309422] acpiphp: Slot [2] registered [ 0.310157] acpiphp: Slot [5] registered [ 0.311149] acpiphp: Slot [6] registered [ 0.312137] acpiphp: Slot [7] registered [ 0.313152] acpiphp: Slot [8] registered [ 0.314135] acpiphp: Slot [9] registered [ 0.315168] acpiphp: Slot [10] registered [ 0.316135] acpiphp: Slot [3] registered [ 0.317102] acpiphp: Slot [4] registered [ 0.318122] acpiphp: Slot [11] registered [ 0.319158] acpiphp: Slot [12] registered [ 0.320115] acpiphp: Slot [13] registered [ 0.321110] acpiphp: Slot [14] registered [ 0.322109] acpiphp: Slot [15] registered [ 0.323116] acpiphp: Slot [16] registered [ 0.324104] acpiphp: Slot [17] registered [ 0.325104] acpiphp: Slot [18] registered [ 0.326121] acpiphp: Slot [19] registered [ 0.327107] acpiphp: Slot [20] registered [ 0.328097] acpiphp: Slot [21] registered [ 0.329108] acpiphp: Slot [22] registered [ 0.330116] acpiphp: Slot [23] registered [ 0.331112] acpiphp: Slot [24] registered [ 0.332142] acpiphp: Slot [25] registered [ 0.333113] acpiphp: Slot [26] registered [ 0.334112] acpiphp: Slot [27] registered [ 0.335104] acpiphp: Slot [28] registered [ 0.336124] acpiphp: Slot [29] registered [ 0.337109] acpiphp: Slot [30] registered [ 0.338113] acpiphp: Slot [31] registered [ 0.339068] PCI host bridge to bus 0000:00 [ 0.340025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.341029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.342029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.343031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.344028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348062] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.350430] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.360051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.361025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.362022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364571] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365806] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.366049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.367767] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.370017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.376019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378715] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.382489] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.385019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.388019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.395032] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.401737] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.404018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.407021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.414020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.421109] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.424018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.427018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.435018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.440905] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.443025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.447021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.454020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.461214] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.493030] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.502024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.526024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.534622] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.545055] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.554022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.570036] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.582897] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.585398] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.587427] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.590489] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.593296] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.598053] iommu: Default domain type: Passthrough [ 0.600419] SCSI subsystem initialized [ 0.601266] ACPI: bus type USB registered [ 0.603114] usbcore: registered new interface driver usbfs [ 0.605100] usbcore: registered new interface driver hub [ 0.606112] usbcore: registered new device driver usb [ 0.607175] pps_core: LinuxPPS API ver. 1 registered [ 0.609017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.611085] PTP clock support registered [ 0.613125] EDAC MC: Ver: 3.0.0 [ 0.614529] PCI: Using ACPI for IRQ routing [ 0.615850] NetLabel: Initializing [ 0.616013] NetLabel: domain hash size = 128 [ 0.617010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.618122] NetLabel: unlabeled traffic allowed by default [ 0.619187] vgaarb: loaded [ 0.621335] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.622014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.627415] clocksource: Switched to clocksource kvm-clock [ 0.728687] VFS: Disk quotas dquot_6.6.0 [ 0.730246] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.732363] *** VALIDATE ramfs *** [ 0.733574] *** VALIDATE hugetlbfs *** [ 0.735067] pnp: PnP ACPI init [ 0.738168] pnp: PnP ACPI: found 6 devices [ 0.754427] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.757545] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.759524] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.761604] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.763613] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.765704] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.768221] NET: Registered protocol family 2 [ 0.770537] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.775124] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.779141] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.784498] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.787918] TCP: Hash tables configured (established 65536 bind 65536) [ 0.790623] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.793628] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.796425] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.799320] NET: Registered protocol family 1 [ 0.803917] RPC: Registered named UNIX socket transport module. [ 0.806025] RPC: Registered udp transport module. [ 0.807675] RPC: Registered tcp transport module. [ 0.809416] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.811436] NET: Registered protocol family 44 [ 0.813195] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.815532] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.817648] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.819908] PCI: CLS 0 bytes, default 64 [ 0.821639] Unpacking initramfs... [ 2.264856] debug: unmapping init [mem 0xffff97ca3cc54000-0xffff97ca3ffbffff] [ 2.268480] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.270947] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.274136] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.785525] Initialise system trusted keyrings [ 2.787069] Key type blacklist registered [ 2.788532] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.799338] zbud: loaded [ 2.802803] *** VALIDATE nfs *** [ 2.804657] *** VALIDATE nfs4 *** [ 2.806467] pstore: using deflate compression [ 2.809883] Platform Keyring initialized [ 2.907742] NET: Registered protocol family 38 [ 2.909760] Key type asymmetric registered [ 2.911344] Asymmetric key parser 'x509' registered [ 2.912700] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.914793] io scheduler mq-deadline registered [ 2.916375] io scheduler kyber registered [ 2.918090] io scheduler bfq registered [ 2.919511] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.921455] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.923942] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.927450] ACPI: Power Button [PWRF] [ 2.933211] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.940304] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.959522] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.969197] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.985898] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.013112] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.039764] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.044470] Non-volatile memory driver v1.3 [ 3.046268] Linux agpgart interface v0.103 [ 3.080657] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.083750] vda: detected capacity change from 0 to 74649600 [ 3.104487] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.107662] vdb: detected capacity change from 0 to 1073741824 [ 3.122125] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.125345] vdc: detected capacity change from 0 to 2621440000 [ 3.139821] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.142575] vdd: detected capacity change from 0 to 2621440000 [ 3.157458] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.160172] vde: detected capacity change from 0 to 4294967296 [ 3.177145] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.180035] vdf: detected capacity change from 0 to 4294967296 [ 3.189707] libphy: Fixed MDIO Bus: probed [ 3.197644] usbcore: registered new interface driver usbserial_generic [ 3.200261] usbserial: USB Serial support registered for generic [ 3.201844] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.205443] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.206660] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.208681] mousedev: PS/2 mouse device common for all mice [ 3.210846] rtc_cmos 00:05: RTC can wake from S4 [ 3.213399] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.213568] rtc_cmos 00:05: registered as rtc0 [ 3.218544] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.220777] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.221600] intel_pstate: CPU model not supported [ 3.225255] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.228449] hid: raw HID events driver (C) Jiri Kosina [ 3.230146] usbcore: registered new interface driver usbhid [ 3.231477] usbhid: USB HID core driver [ 3.232420] drop_monitor: Initializing network drop monitor service [ 3.234030] Initializing XFRM netlink socket [ 3.235773] NET: Registered protocol family 10 [ 3.238577] Segment Routing with IPv6 [ 3.240097] NET: Registered protocol family 17 [ 3.242185] mpls_gso: MPLS GSO support [ 3.248045] RAS: Correctable Errors collector initialized. [ 3.249637] AVX version of gcm_enc/dec engaged. [ 3.251320] AES CTR mode by8 optimization enabled [ 3.323516] sched_clock: Marking stable (3323480166, 0)->(4403331412, -1079851246) [ 3.327584] registered taskstats version 1 [ 3.329720] Loading compiled-in X.509 certificates [ 3.331273] zswap: loaded using pool lzo/zbud [ 3.355751] Key type big_key registered [ 3.368890] Key type encrypted registered [ 3.370478] ima: No TPM chip found, activating TPM-bypass! [ 3.372316] ima: Allocated hash algorithm: sha1 [ 3.374603] ima: No architecture policies found [ 3.376136] evm: Initialising EVM extended attributes: [ 3.378015] evm: security.selinux [ 3.379708] evm: security.ima [ 3.381337] evm: security.capability [ 3.383658] evm: HMAC attrs: 0x1 [ 3.386307] rtc_cmos 00:05: setting system clock to 2026-07-26 18:40:10 UTC (1785091210) [ 3.392814] debug: unmapping init [mem 0xffffffffb7a03000-0xffffffffb7bfffff] [ 3.396282] debug: unmapping init [mem 0xffffffffb6782000-0xffffffffb6a58fff] [ 3.408102] Write protecting the kernel read-only data: 28672k [ 3.411344] debug: unmapping init [mem 0xffffffffb4e03000-0xffffffffb4ffffff] [ 3.413398] debug: unmapping init [mem 0xffffffffb5714000-0xffffffffb57fffff] [ 3.446538] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.453298] systemd[1]: Detected virtualization kvm. [ 3.455248] systemd[1]: Detected architecture x86-64. [ 3.456959] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.483594] systemd[1]: No hostname configured. [ 3.485212] systemd[1]: Set hostname to . [ 3.487205] random: systemd: uninitialized urandom read (16 bytes read) [ 3.489534] systemd[1]: Initializing machine ID from random generator. [ 3.540795] random: ln: uninitialized urandom read (6 bytes read) [ 3.645325] random: systemd: uninitialized urandom read (16 bytes read) [ 3.647377] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.651355] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.655457] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.330512] device-mapper: uevent: version 1.0.3 [ 4.332410] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.055397] virtio_net virtio0 ens2: renamed from eth0 [ 5.091780] random: fast init done [ 5.267371] scsi host0: ata_piix [ 5.285450] scsi host1: ata_piix [ 5.286983] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.289261] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.910349] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.894386] random: crng init done [ 9.896129] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.401270] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.630503] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.910759] SELinux: Disabled at runtime. [ 11.975499] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.983339] systemd[1]: Detected virtualization kvm. [ 11.985266] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.471634] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.475168] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.480340] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.485379] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.489830] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.496214] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.510042] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ 12.645455] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.940555] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.233329] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.251322] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.391585] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.407885] EDAC sbridge: Ver: 1.1.2 [ 14.624517] Key type dns_resolver registered [ 14.932913] NFS: Registering the id_resolver key type [ 14.934421] Key type id_resolver registered [ 14.935859] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg120-server login: [ 40.691279] libcfs: loading out-of-tree module taints kernel. [ 40.711219] Key type ._llcrypt registered [ 40.712936] Key type .llcrypt registered [ 40.766294] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 49.822947] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 50.568261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.576685] alg: No test for adler32 (adler32-zlib) [ 51.600520] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 51.950358] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 53.592195] Key type lgssc registered [ 54.314780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.468935] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 110.915298] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 124.381702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 124.424380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 125.797595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 125.831367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 125.917798] Lustre: lustre-MDT0000: new disk, initializing [ 126.004838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.038848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 129.866140] hrtimer: interrupt took 8849677 ns [ 131.414584] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.184730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.285500] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 144.313304] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 144.317821] Lustre: Skipped 1 previous similar message [ 144.411846] Lustre: lustre-MDT0001: new disk, initializing [ 144.494470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 144.527036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 144.540560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 148.980351] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.145683] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 164.429258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 164.750970] Lustre: lustre-OST0000: new disk, initializing [ 164.753865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 164.758457] Lustre: 8420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 164.850306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.024742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 171.033762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 171.133599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 171.308681] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.412814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 187.642524] Lustre: lustre-OST0001: new disk, initializing [ 187.654535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 187.666966] Lustre: 9491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 187.799837] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.130943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 195.142151] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 195.206087] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 196.667894] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.326962] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 218.043900] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 226.966798] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing check_logdir /tmp/testlogs/ [ 232.883544] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing yml_node [ 239.978978] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 242.812839] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 245.514673] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 247.091323] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Jul 26 14:44:12 EDT 2026 [ 264.174553] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 266.123796] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 282.599653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 282.615929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.642985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.201956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.210915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.239183] Lustre: Skipped 1 previous similar message [ 288.789768] Lustre: server umount lustre-MDT0000 complete [ 292.883043] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785091499 with bad export cookie 6997025511758722305 [ 292.886521] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.902352] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 297.441208] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.445544] LustreError: 10445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.448220] Lustre: Skipped 1 previous similar message [ 297.453427] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.465059] LustreError: 10445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 297.482932] Lustre: Skipped 4 previous similar messages [ 299.271583] Lustre: server umount lustre-MDT0001 complete [ 303.345695] Lustre: server umount lustre-OST0000 complete [ 308.759469] Lustre: server umount lustre-OST0001 complete [ 317.120854] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 327.598775] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 370.677082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.957843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 371.032295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 371.102237] Lustre: lustre-MDT0000: new disk, initializing [ 371.236052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.288602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 376.641729] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 387.475610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.626894] Lustre: 15052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 387.707042] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 387.710438] Lustre: Skipped 1 previous similar message [ 387.822682] Lustre: lustre-MDT0001: new disk, initializing [ 387.994528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 388.039497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 388.057866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 392.993334] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 402.672902] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 408.839698] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 414.958889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 415.247908] Lustre: lustre-OST0000: new disk, initializing [ 415.253855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 415.260592] Lustre: 17020:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 415.337685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 417.269125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 417.277208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 417.395490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 421.585051] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 431.118921] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 435.801862] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 436.013248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 440.730697] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 440.937540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 447.970120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.974739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 447.984295] Lustre: Skipped 2 previous similar messages [ 449.297522] Lustre: server umount lustre-OST0000 complete [ 453.090364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.092129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 453.112515] Lustre: Skipped 1 previous similar message [ 453.121388] Lustre: Skipped 2 previous similar messages [ 459.393682] Lustre: server umount lustre-MDT0000 complete [ 463.261417] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785091670 with bad export cookie 6997025511758723838 [ 463.262581] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 463.298857] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 463.634694] Lustre: server umount lustre-MDT0001 complete [ 470.450672] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:47:55 (1785091675) [ 488.888724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.440405] LustreError: 19895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.535407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.165122] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 494.561829] LustreError: 19896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.682230] LustreError: 19895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.359527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 507.614229] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 518.080270] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 525.808140] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 527.956462] Lustre: Failing over lustre-MDT0000 [ 528.164883] Lustre: server umount lustre-MDT0000 complete [ 528.998415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.014149] LustreError: 20623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.039074] LustreError: 20623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 539.619932] LustreError: 19891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.649893] LustreError: 19891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 545.768243] Lustre: 3618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785091736/real 1785091736] req@ffff97c98375c000 x1871803852162560/t0(0) o400->MGC192.168.201.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785091752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 545.802368] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.598157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.055391] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x611a6da190b98084 [ 555.079779] Lustre: MGC192.168.201.120@tcp: Connection restored to 0@lo (at 0@lo) [ 555.334951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.344719] Lustre: Skipped 1 previous similar message [ 559.737946] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 560.624985] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 560.632296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 560.652656] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 569.887111] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 575.987086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 576.003215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.029883] Lustre: Skipped 1 previous similar message [ 576.037383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 576.047076] Lustre: Skipped 2 previous similar messages [ 580.624604] Lustre: server umount lustre-MDT0000 complete [ 584.172610] LustreError: 19890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.213596] LustreError: 19890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 585.042018] LustreError: 19877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785091792 with bad export cookie 6997025511758725252 [ 585.042822] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 585.052146] LustreError: 19877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 585.726364] Lustre: server umount lustre-MDT0001 complete [ 596.234815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.327576] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 611.125995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.268411] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 624.067804] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 629.683283] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 631.898872] Lustre: Failing over lustre-MDT0000 [ 632.117760] Lustre: server umount lustre-MDT0000 complete [ 632.303734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 632.313637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.325647] Lustre: Skipped 1 previous similar message [ 632.337941] LustreError: 23233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.358263] LustreError: 23233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 652.112454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 652.214455] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 652.471844] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 656.476227] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 657.578447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 657.579035] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 657.582966] Lustre: Skipped 1 previous similar message [ 657.612364] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 667.097629] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 673.285750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 673.292450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 673.300393] Lustre: Skipped 3 previous similar messages [ 677.915590] Lustre: server umount lustre-MDT0000 complete [ 681.673621] LustreError: 23214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785091888 with bad export cookie 6997025511758726197 [ 681.679037] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.689685] LustreError: 23214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 682.195686] Lustre: server umount lustre-MDT0001 complete [ 691.416697] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:51:37 (1785091897) [ 706.283632] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 717.824990] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 760.047628] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 771.577116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 771.949596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 772.052236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 772.198268] Lustre: lustre-MDT0000: new disk, initializing [ 772.321761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 772.327393] Lustre: Skipped 3 previous similar messages [ 772.349940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 777.836904] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 789.304894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.440656] Lustre: 29606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 789.819437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 795.302222] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 804.959804] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 810.995551] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 817.958759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 818.357162] Lustre: 31582:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 820.256863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 820.283690] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 820.342249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 820.364611] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 825.547586] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 836.329411] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 842.879694] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 861.667481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 861.679319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.703923] Lustre: Skipped 3 previous similar messages [ 861.713497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 867.857894] Lustre: server umount lustre-OST0000 complete [ 871.905452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 871.915772] LustreError: Skipped 1 previous similar message [ 878.054501] Lustre: server umount lustre-MDT0000 complete [ 881.641887] LustreError: 30487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.656117] LustreError: 30487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 882.527219] LustreError: 29601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785092089 with bad export cookie 6997025511758726603 [ 882.540636] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 882.548102] LustreError: 29601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 882.940584] Lustre: server umount lustre-MDT0001 complete [ 895.509905] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 899.089600] Key type lgssc unregistered [ 899.422806] LNet: 33554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.437228] LNetError: 33554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.456635] LNet: Removed LNI 192.168.201.120@tcp [ 900.447162] Key type .llcrypt unregistered [ 900.449647] Key type ._llcrypt unregistered [ 928.097945] Key type ._llcrypt registered [ 928.100542] Key type .llcrypt registered [ 928.275489] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 929.344074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 929.517436] alg: No test for adler32 (adler32-zlib) [ 930.926512] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 931.182172] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 932.896180] Key type lgssc registered [ 934.393514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 942.190145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 942.247868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.752385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 949.048167] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 953.314456] LustreError: 34969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.346671] LustreError: 34969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 958.444878] LustreError: 34970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.579249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.061819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 963.560641] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 971.217381] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 977.642826] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 984.961192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.327734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.437614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 992.170162] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1000.647507] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1004.785745] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1017.967858] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:57:03 (1785092223) [ 1020.219651] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1022.410295] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1030.185454] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:57:15 (1785092235) [ 1038.985726] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:57:24 (1785092244) [ 1040.141793] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1048.683760] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1048.689956] Lustre: Skipped 2 previous similar messages [ 1050.374490] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:57:36 (1785092256) [ 1053.978585] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1053.980881] Lustre: Skipped 3 previous similar messages [ 1073.127428] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1073.140746] Lustre: Skipped 6 previous similar messages [ 1077.002508] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:58:02 (1785092282) [ 1079.638169] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1082.934522] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1091.118324] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:58:16 (1785092296) [ 1092.839356] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1092.844804] Lustre: Skipped 3 previous similar messages [ 1105.854066] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:58:31 (1785092311) [ 1121.771495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1121.793719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.817788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1122.791580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1122.792461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1122.799144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.805240] Lustre: Skipped 1 previous similar message [ 1125.464732] Lustre: server umount lustre-OST0000 complete [ 1133.025353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1133.026610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.033461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1133.047017] Lustre: Skipped 1 previous similar message [ 1135.591679] Lustre: server umount lustre-MDT0000 complete [ 1138.145977] LustreError: 35704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.164508] LustreError: 35704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1139.422355] LustreError: 36815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785092346 with bad export cookie 13894646203757470422 [ 1139.427544] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1139.432362] LustreError: 36815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1139.719277] Lustre: server umount lustre-MDT0001 complete [ 1151.702371] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1154.682661] Key type lgssc unregistered [ 1154.876044] LNet: 42817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.884300] LNetError: 42817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.898830] LNet: Removed LNI 192.168.201.120@tcp [ 1155.768730] Key type .llcrypt unregistered [ 1155.771130] Key type ._llcrypt unregistered [ 1174.541545] Key type ._llcrypt registered [ 1174.544728] Key type .llcrypt registered [ 1174.653234] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 1191.330556] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1192.747741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1192.838111] alg: No test for adler32 (adler32-zlib) [ 1193.956339] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1194.323735] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1196.032306] Key type lgssc registered [ 1197.156899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1229.420936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1229.463146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.680180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1230.698371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1230.813620] Lustre: lustre-MDT0000: new disk, initializing [ 1230.887172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.913567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1235.558565] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1245.687968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.799229] Lustre: 45969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1245.840842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1245.844199] Lustre: Skipped 1 previous similar message [ 1245.896865] Lustre: lustre-MDT0001: new disk, initializing [ 1245.989669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1246.022082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1246.034680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1251.447592] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1260.679196] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1266.146375] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1272.323150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1272.601683] Lustre: lustre-OST0000: new disk, initializing [ 1272.604951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1272.611107] Lustre: 47950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1272.687896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.666572] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1281.614315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1281.631096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1281.783091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1288.704905] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1293.313546] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1293.538609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1298.333373] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1298.536200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1302.499782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.503505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1302.511175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1302.516262] Lustre: Skipped 1 previous similar message [ 1302.532889] Lustre: Skipped 1 previous similar message [ 1306.543077] Lustre: server umount lustre-OST0000 complete [ 1312.738273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1312.738799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1312.756715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1312.774155] Lustre: Skipped 1 previous similar message [ 1316.314332] Lustre: server umount lustre-MDT0000 complete [ 1319.706310] LustreError: 45962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785092526 with bad export cookie 7127594476013305249 [ 1319.708441] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1319.719433] LustreError: 45962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1319.972773] Lustre: server umount lustre-MDT0001 complete [ 1329.125638] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:02:14 (1785092534) [ 1336.522618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.046776] LustreError: 50413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.073054] LustreError: 50413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1337.165061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.467244] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1342.434895] LustreError: 50414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.553558] LustreError: 50413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.341914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.798474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1354.404574] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1361.950228] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1367.673511] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1374.303907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1374.646091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.030231] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1388.173695] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1392.970876] LustreError: 52941:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1393.767666] Lustre: Modifying parameter general.timeout=1 in log params [ 1395.454138] Lustre: Modifying parameter general.timeout=3 in log params [ 1395.459174] Lustre: Skipped 1 previous similar message [ 1397.824535] Lustre: Modifying parameter general.timeout=6 in log params [ 1397.832833] Lustre: Skipped 2 previous similar messages [ 1402.227703] Lustre: Modifying parameter general.timeout=11 in log params [ 1402.230572] Lustre: Skipped 4 previous similar messages [ 1410.503166] Lustre: Modifying parameter general.timeout=21 in log params [ 1410.507605] Lustre: Skipped 9 previous similar messages [ 1426.799055] Lustre: Modifying parameter general.timeout=41 in log params [ 1426.805064] Lustre: Skipped 19 previous similar messages [ 1459.658421] Lustre: Modifying parameter general.timeout=81 in log params [ 1459.663636] Lustre: Skipped 39 previous similar messages [ 1508.832289] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:05:14 (1785092714) [ 1511.210742] Lustre: 59875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1512.802964] Lustre: 59973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1516.173208] Lustre: 60180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1516.184250] Lustre: 60180:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1523.947797] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1523.962811] Lustre: Skipped 58 previous similar messages [ 1542.908462] Lustre: 61744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1542.917387] Lustre: 61744:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1542.930606] LustreError: 61744:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1546.137991] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:05:52 (1785092752) [ 1559.526974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1559.536908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1559.557633] Lustre: Skipped 1 previous similar message [ 1559.566524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1563.900260] Lustre: server umount lustre-MDT0000 complete [ 1565.670747] LustreError: 50413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1565.688584] LustreError: 50413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1566.957623] LustreError: 52990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785092774 with bad export cookie 7127594476013306131 [ 1566.959519] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1566.964725] LustreError: 52990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1567.221921] Lustre: server umount lustre-MDT0001 complete [ 1580.216628] Lustre: server umount lustre-OST0000 complete [ 1602.603406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1602.954938] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1603.180957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.483230] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1615.120810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.191797] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1615.237925] Lustre: 64278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1620.101808] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1626.225534] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1630.859888] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1637.123060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1637.303517] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1637.593866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.607375] Lustre: Skipped 1 previous similar message [ 1644.175267] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1651.879405] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1654.064128] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1654.069427] Lustre: Skipped 13 previous similar messages [ 1654.126119] Lustre: 66845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1664.481557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1664.484963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.516196] Lustre: Skipped 2 previous similar messages [ 1664.522056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1664.529158] Lustre: Skipped 3 previous similar messages [ 1667.006214] Lustre: server umount lustre-OST0000 complete [ 1672.161965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1672.168109] LustreError: Skipped 1 previous similar message [ 1672.170160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.181273] Lustre: Skipped 1 previous similar message [ 1672.188082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.190803] Lustre: Skipped 1 previous similar message [ 1676.712868] Lustre: server umount lustre-MDT0000 complete [ 1680.070914] LustreError: 64271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785092887 with bad export cookie 7127594476013309092 [ 1680.079855] LustreError: 64271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1680.081299] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1680.345903] Lustre: server umount lustre-MDT0001 complete [ 1691.090956] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1693.606565] Key type lgssc unregistered [ 1694.002628] LNet: 68052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.009887] LNetError: 68052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.094166] LNet: Removed LNI 192.168.201.120@tcp [ 1695.779899] Key type .llcrypt unregistered [ 1695.782951] Key type ._llcrypt unregistered [ 1717.249780] Key type ._llcrypt registered [ 1717.252275] Key type .llcrypt registered [ 1717.340473] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1718.056651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1718.160875] alg: No test for adler32 (adler32-zlib) [ 1719.231936] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1719.472627] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1721.160218] Key type lgssc registered [ 1722.108842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1728.846857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1728.866236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.362341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.267446] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1740.774092] LustreError: 69466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1740.812691] LustreError: 69466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1741.629273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.101600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1746.510830] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1752.421937] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1757.327958] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1762.766708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1763.183094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.647812] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1774.866424] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1783.608299] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1790.952159] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.953707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.961389] Lustre: Skipped 1 previous similar message [ 1790.971896] Lustre: Skipped 2 previous similar messages [ 1796.067358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1796.076111] Lustre: Skipped 1 previous similar message [ 1797.252364] Lustre: server umount lustre-MDT0000 complete [ 1800.374392] LustreError: 69447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785093007 with bad export cookie 14212927761025495431 [ 1800.382053] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1800.385238] LustreError: 69447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1800.688591] Lustre: server umount lustre-MDT0001 complete [ 1813.197631] Lustre: server umount lustre-OST0000 complete [ 1837.393672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.708912] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1837.749481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1838.050632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.508799] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1849.808368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.873898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1849.893643] Lustre: 74334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1850.237976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1854.011486] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1860.971437] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1865.781826] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1871.454759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1871.565904] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1871.892285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.645420] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1882.505284] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1883.859775] Lustre: 76905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1885.316466] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1890.785680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.790356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.803021] Lustre: Skipped 1 previous similar message [ 1890.810468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1891.314061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1895.824685] Lustre: server umount lustre-OST0000 complete [ 1901.538856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.541858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1901.550514] Lustre: Skipped 2 previous similar messages [ 1901.554154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.570341] Lustre: Skipped 3 previous similar messages [ 1905.631947] Lustre: server umount lustre-MDT0000 complete [ 1906.659834] LustreError: 74344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.682252] LustreError: 74344:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1908.542260] LustreError: 74325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785093115 with bad export cookie 14212927761025496257 [ 1908.547225] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1908.553350] LustreError: 74325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1908.870375] Lustre: server umount lustre-MDT0001 complete [ 1917.073436] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1918.933579] Key type lgssc unregistered [ 1919.219278] LNet: 78113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.228178] LNetError: 78113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.248209] LNet: Removed LNI 192.168.201.120@tcp [ 1919.911137] Key type .llcrypt unregistered [ 1919.913053] Key type ._llcrypt unregistered [ 1939.789454] Key type ._llcrypt registered [ 1939.795225] Key type .llcrypt registered [ 1939.908852] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1940.577138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1940.677155] alg: No test for adler32 (adler32-zlib) [ 1941.695355] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1941.905077] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1943.520179] Key type lgssc registered [ 1944.399243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1950.454976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1950.481063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.880642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.532630] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1961.456394] LustreError: 79524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.473532] LustreError: 79524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1962.542381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.997761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1966.518532] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1972.740195] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1977.081577] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1982.587897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.991846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.883823] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1994.079472] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2004.720916] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2005.489643] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2007.233477] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2007.238927] Lustre: Skipped 1 previous similar message [ 2009.318836] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2009.324429] Lustre: Skipped 2 previous similar messages [ 2013.456563] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2013.459045] Lustre: Skipped 5 previous similar messages [ 2021.470800] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2021.473157] Lustre: Skipped 11 previous similar messages [ 2038.184149] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2038.187251] Lustre: Skipped 23 previous similar messages [ 2070.680780] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2070.688044] Lustre: Skipped 43 previous similar messages [ 2082.408168] Lustre: 87612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2084.414746] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:14:50 (1785093290) [ 2085.638952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2087.205845] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:14:53 (1785093293) [ 2093.544086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2093.552703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.563612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2093.579986] Lustre: Skipped 1 previous similar message [ 2095.812698] Lustre: server umount lustre-OST0000 complete [ 2098.660930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.669571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.671986] Lustre: Skipped 1 previous similar message [ 2103.778599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.793150] Lustre: Skipped 1 previous similar message [ 2104.815554] Lustre: server umount lustre-MDT0000 complete [ 2108.131630] LustreError: 79505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785093315 with bad export cookie 14979651038922893177 [ 2108.131955] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2108.139839] LustreError: 79505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2108.377703] Lustre: server umount lustre-MDT0001 complete [ 2115.939391] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2117.717587] Key type lgssc unregistered [ 2117.926690] LNet: 88859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.933306] LNetError: 88859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.946963] LNet: Removed LNI 192.168.201.120@tcp [ 2118.430206] Key type .llcrypt unregistered [ 2118.432436] Key type ._llcrypt unregistered [ 2120.808961] Key type ._llcrypt registered [ 2120.810550] Key type .llcrypt registered [ 2120.894265] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2124.078133] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules [ 2124.699766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2124.814221] alg: No test for adler32 (adler32-zlib) [ 2125.757922] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2125.917933] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2127.560125] Key type lgssc registered [ 2128.263408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2129.075326] LustreError: 90062:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2141.883420] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2148.506593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2151.003354] LustreError: 90062:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2151.058217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.452817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.333516] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2160.944598] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:16:06 (1785093366) [ 2162.665286] LustreError: 91086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.684388] LustreError: 91086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2164.617814] LustreError: 91117:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2164.623635] LustreError: 91117:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2164.627809] LustreError: 91117:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 2164.871047] Lustre: server umount lustre-MDT0000 complete [ 2175.427238] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2177.361588] Key type lgssc unregistered [ 2177.552909] LNet: 92611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2177.556183] LNetError: 92611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2177.571727] LNet: Removed LNI 192.168.201.120@tcp [ 2178.120834] Key type .llcrypt unregistered [ 2178.123274] Key type ._llcrypt unregistered [ 2191.044851] Key type ._llcrypt registered [ 2191.047952] Key type .llcrypt registered [ 2191.137302] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2191.912864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2191.999323] alg: No test for adler32 (adler32-zlib) [ 2192.974037] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2193.126106] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2194.728218] Key type lgssc registered [ 2195.496693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2200.408901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2200.423851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.701840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.113419] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2209.763612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.186027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2212.916685] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2218.700900] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2222.876881] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2227.345182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2227.759332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.679130] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2236.699234] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2282.069345] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:18:07 (1785093487) [ 2283.362266] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2284.477416] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:18:10 (1785093490) [ 2292.193668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2292.197582] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.201750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.213121] Lustre: Skipped 2 previous similar messages [ 2292.218783] Lustre: Skipped 1 previous similar message [ 2293.437226] Lustre: server umount lustre-MDT0000 complete [ 2295.601086] LustreError: 93787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785093502 with bad export cookie 17736646821731547652 [ 2295.606103] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.609280] LustreError: 93787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2295.798161] Lustre: server umount lustre-MDT0001 complete [ 2308.398649] Lustre: server umount lustre-OST0000 complete [ 2315.621333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.864794] LustreError: 98098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.882738] LustreError: 98098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2315.971793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.100918] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2321.376944] LustreError: 98099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.121248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.499882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2326.489379] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2330.746995] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2334.569922] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2342.156735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2342.272778] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2342.278897] LustreError: 98088:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2342.298398] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2342.302676] LustreError: 100181:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2342.306775] LustreError: 100181:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2342.310690] LustreError: 100181:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2342.360658] Lustre: server umount lustre-OST0000 complete [ 2342.363160] LustreError: 100181:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2347.200579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2347.299978] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2347.309552] LustreError: 98088:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2347.317779] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2347.326775] LustreError: 100641:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2347.336264] LustreError: 100641:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2347.343523] LustreError: 100641:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2347.401896] Lustre: server umount lustre-OST0000 complete [ 2347.405730] LustreError: 100641:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2352.763469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2352.876743] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2352.880384] Lustre: Found index 0 for lustre-OST0000, updating log [ 2352.884350] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2352.949700] Lustre: lustre-OST0000: new disk, initializing [ 2352.953417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2352.958758] Lustre: 101176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2353.148355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2356.648154] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2358.672325] Lustre: Failing over lustre-OST0000 [ 2358.774463] Lustre: server umount lustre-OST0000 complete [ 2364.385537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2364.389162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.397447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2365.921040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2365.923361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.279722] Lustre: server umount lustre-MDT0000 complete [ 2367.969593] LustreError: 98098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2367.992569] LustreError: 98098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2369.623850] LustreError: 98078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785093576 with bad export cookie 17736646821731550382 [ 2369.624941] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2369.632842] LustreError: 98078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2369.857899] Lustre: server umount lustre-MDT0001 complete [ 2374.780669] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:19:41 (1785093581) [ 2384.025164] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2390.541805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.767712] LustreError: 103516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.826307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.001524] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2396.131972] LustreError: 103517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.952647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.831593] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2402.450495] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2406.286028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2410.213376] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2414.696288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2414.751637] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2414.802614] Lustre: lustre-OST0001: new disk, initializing [ 2414.805162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2414.808902] Lustre: 105832:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2418.477241] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2423.155879] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2423.324625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2423.331114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2423.376447] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2429.286941] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2433.515203] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2433.519703] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2433.527727] Lustre: Skipped 2 previous similar messages [ 2433.537402] Lustre: Skipped 3 previous similar messages [ 2438.626304] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2438.628397] Lustre: Skipped 1 previous similar message [ 2446.304133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2446.504494] Lustre: server umount lustre-MDT0001 complete [ 2448.868325] LustreError: 103512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.883070] LustreError: 103512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2452.451853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.531850] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2452.533959] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2452.561442] Lustre: 103504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2452.990650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2452.994362] Lustre: Skipped 3 previous similar messages [ 2454.507223] LustreError: 107462:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2454.513258] LustreError: 107462:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2454.519880] LustreError: 103508:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2454.570118] LustreError: 107482:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.120@tcp: cfg command failed: rc = -17 [ 2454.580101] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2454.580101] [ 2455.501274] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2458.102245] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2460.935566] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 15:21:07 (1785093667) [ 2470.860970] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2481.774261] Lustre: 109452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2495.318378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2497.389808] Lustre: 110586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2565.600948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2565.602532] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.604229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.604235] Lustre: Skipped 5 previous similar messages [ 2565.621564] Lustre: Skipped 2 previous similar messages [ 2567.255797] Lustre: server umount lustre-MDT0000 complete [ 2569.378176] LustreError: 103497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785093776 with bad export cookie 17736646821731551299 [ 2569.380983] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2569.383631] LustreError: 103497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2569.600396] Lustre: server umount lustre-MDT0001 complete [ 2582.222919] Lustre: server umount lustre-OST0000 complete [ 2594.200366] Lustre: server umount lustre-OST0001 complete [ 2597.712081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2603.287253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2608.505466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2614.953731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2627.266896] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2633.662319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.681306] Lustre: lustre-MDT0000: reset Object Index mappings [ 2633.990777] LustreError: 114533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.996810] LustreError: 114533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2634.056718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.964209] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2642.856438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.874242] Lustre: lustre-MDT0001: reset Object Index mappings [ 2646.007875] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2647.748231] Lustre: 115679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2651.550370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.040345] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 2655.381222] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2659.320312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 2660.833269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2665.224174] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2666.476568] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2666.477787] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2670.695528] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2672.853316] Lustre: 117554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2703.528106] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 15:25:09 (1785093909) [ 2733.025751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.029677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2733.031096] Lustre: Skipped 3 previous similar messages [ 2733.036178] Lustre: Skipped 6 previous similar messages [ 2737.818286] Lustre: server umount lustre-MDT0000 complete [ 2738.152877] LustreError: 114533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.167769] LustreError: 114533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 2739.992974] LustreError: 114515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785093947 with bad export cookie 17736646821731598129 [ 2739.994990] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2740.002506] LustreError: 114515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2746.265995] Lustre: server umount lustre-MDT0001 complete [ 2754.283290] Lustre: server umount lustre-OST0000 complete [ 2762.564616] Lustre: server umount lustre-OST0001 complete [ 2766.493783] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 2772.234205] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2793.187534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2793.351846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2793.366111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2793.425077] Lustre: lustre-MDT0000: new disk, initializing [ 2793.468458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2793.475200] Lustre: Skipped 3 previous similar messages [ 2793.486656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2796.198949] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2800.967605] Lustre: server umount lustre-MDT0000 complete [ 2806.418256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.624921] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2806.639565] Lustre: 121818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2806.648878] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2809.367713] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2812.281889] Lustre: server umount lustre-MDT0000 complete [ 2816.807381] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:27:03 (1785094023) [ 2817.800332] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2818.900502] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:27:05 (1785094025) [ 2827.745483] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2832.869925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.339535] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2839.410960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.473640] Lustre: 123868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2839.508216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2839.510203] Lustre: Skipped 1 previous similar message [ 2839.552890] Lustre: lustre-MDT0001: new disk, initializing [ 2839.636342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2839.644503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2841.680980] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2844.885919] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2848.255564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2848.434073] Lustre: lustre-OST0000: new disk, initializing [ 2848.436420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2848.439639] Lustre: 125444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2848.532058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2848.536810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2848.619065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2851.443154] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2857.712727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2857.794774] Lustre: lustre-OST0001: new disk, initializing [ 2857.797094] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2857.800474] Lustre: 126313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2859.735501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2859.739338] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2859.796415] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2861.380923] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2867.840267] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2869.492304] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3080.730034] Lustre: 132701:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3080.734872] Lustre: 132701:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3080.739524] Lustre: 132701:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3080.743930] Lustre: 132701:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3080.747708] Lustre: 132701:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3080.751682] Lustre: 132701:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3237.351884] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:34:03 (1785094443) [ 3249.172758] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3249.182544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.193627] Lustre: Skipped 3 previous similar messages [ 3249.197028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3249.199214] Lustre: Skipped 4 previous similar messages [ 3255.470963] Lustre: server umount lustre-MDT0000 complete [ 3257.293496] LustreError: 132716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785094464 with bad export cookie 17736646821731659925 [ 3257.299431] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3257.301552] LustreError: 132716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3257.314318] LustreError: 123875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.323367] LustreError: 123875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3263.671954] Lustre: server umount lustre-MDT0001 complete [ 3280.352136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3280.394838] Lustre: server umount lustre-OST0000 complete [ 3288.201720] Lustre: server umount lustre-OST0001 complete [ 3290.855241] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 3295.136600] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3313.464263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3313.621675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3313.646646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3313.694288] Lustre: lustre-MDT0000: new disk, initializing [ 3313.742166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3313.746054] Lustre: Skipped 5 previous similar messages [ 3313.753807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3315.830033] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3321.028183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.077996] Lustre: 135731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3321.225928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3322.971298] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3328.036988] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3330.834395] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3333.684922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3333.836285] Lustre: 137699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3335.824158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3335.828390] Lustre: Skipped 1 previous similar message [ 3335.830584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3335.881170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3336.403886] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3340.875926] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3345.005649] Lustre: lustre-MDD0000: changelog on [ 3346.098870] Lustre: lustre-MDD0001: changelog on [ 4071.667609] Lustre: lustre-MDD0001: changelog off [ 4072.405893] Lustre: 135737:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4072.734160] Lustre: lustre-MDD0000: changelog off [ 4076.525228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4076.527605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4076.531922] Lustre: Skipped 6 previous similar messages [ 4076.538826] Lustre: Skipped 9 previous similar messages [ 4080.806895] Lustre: server umount lustre-OST0000 complete [ 4084.193228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4084.196274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.753830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4086.756126] Lustre: Skipped 2 previous similar messages [ 4088.659252] Lustre: server umount lustre-MDT0000 complete [ 4090.269392] LustreError: 135723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785095297 with bad export cookie 17736646821732471925 [ 4090.272694] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4090.274423] LustreError: 135723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4090.411101] Lustre: server umount lustre-MDT0001 complete [ 4094.700428] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 4095.837454] Key type lgssc unregistered [ 4095.991547] LNet: 146602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4095.998654] LNetError: 146602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4096.007408] LNet: Removed LNI 192.168.201.120@tcp [ 4096.390175] Key type .llcrypt unregistered [ 4096.392231] Key type ._llcrypt unregistered [ 4105.203727] Key type ._llcrypt registered [ 4105.205187] Key type .llcrypt registered [ 4105.249619] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4112.751340] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4113.217051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4113.229201] alg: No test for adler32 (adler32-zlib) [ 4114.090776] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 4114.182715] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 4115.760264] Key type lgssc registered [ 4116.186959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4129.444922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4129.450922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4130.533311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4130.545242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4130.580448] Lustre: lustre-MDT0000: new disk, initializing [ 4130.605274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4130.613603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4132.105236] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4136.399364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.438657] Lustre: 149775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4136.467766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4136.469938] Lustre: Skipped 1 previous similar message [ 4136.505564] Lustre: lustre-MDT0001: new disk, initializing [ 4136.539891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4136.552348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4136.556599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4137.931899] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4141.754626] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4143.669700] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4146.306348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4146.411532] Lustre: lustre-OST0000: new disk, initializing [ 4146.413373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4146.416078] Lustre: 151745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4146.444288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.652845] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4152.931926] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4156.407468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4156.411055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4156.441824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4160.314861] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_module obdecho/obdecho [ 4160.941348] LustreError: 152749:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4160.944058] LustreError: 152749:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4161.550601] LustreError: 152845:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 4166.631270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.632081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4166.637364] Lustre: Skipped 2 previous similar messages [ 4166.638870] Lustre: Skipped 1 previous similar message [ 4171.029636] Lustre: server umount lustre-MDT0000 complete [ 4172.370105] LustreError: 149769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785095379 with bad export cookie 8079579724305390728 [ 4172.373316] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4172.375024] LustreError: 149769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4172.490955] Lustre: server umount lustre-MDT0001 complete [ 4184.437509] Lustre: server umount lustre-OST0000 complete [ 4192.241324] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4195.998594] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4210.059520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.182882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4210.202761] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4210.245616] Lustre: lustre-MDT0000: new disk, initializing [ 4210.275303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4210.282321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4211.835033] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4216.500296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.549880] Lustre: 156550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4216.576583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4216.578732] Lustre: Skipped 1 previous similar message [ 4216.618092] Lustre: lustre-MDT0001: new disk, initializing [ 4216.664936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4216.680484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4216.685224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4218.337673] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4222.488961] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4224.515525] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4227.176078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4227.280790] Lustre: lustre-OST0000: new disk, initializing [ 4227.282644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4227.285033] Lustre: 158519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4227.318773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4229.489710] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4232.705867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4232.709667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4232.737704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4233.641699] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4235.198640] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4235.276087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4236.830505] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4236.917428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4237.793673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4237.796421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.804587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4237.807394] Lustre: Skipped 2 previous similar messages [ 4239.698955] Lustre: server umount lustre-OST0000 complete [ 4242.401958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4242.405663] LustreError: Skipped 1 previous similar message [ 4242.407751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.412437] Lustre: Skipped 1 previous similar message [ 4242.414950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4247.134766] Lustre: server umount lustre-MDT0000 complete [ 4248.033292] LustreError: 156562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4248.040187] LustreError: 156562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4248.552456] LustreError: 156544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785095455 with bad export cookie 8079579724305391673 [ 4248.555603] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4248.557697] LustreError: 156544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4248.714220] Lustre: server umount lustre-MDT0001 complete [ 4250.077483] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:50:56 (1785095456) [ 4250.641296] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4251.355416] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:50:57 (1785095457) [ 4254.138042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.312092] LustreError: 160924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.336856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.775225] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4258.749879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4260.335434] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4263.048372] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4265.194604] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4267.896278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4268.003311] LustreError: 162742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4270.472240] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4273.322235] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4288.764298] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 4288.767772] Lustre: Skipped 1 previous similar message [ 4289.505882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4289.509800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.515840] Lustre: Skipped 1 previous similar message [ 4292.897946] Lustre: server umount lustre-MDT0000 complete [ 4293.867679] LustreError: 160920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4294.660126] LustreError: 160904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785095501 with bad export cookie 8079579724305392541 [ 4294.663320] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.664484] LustreError: 160904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4295.656925] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4295.665349] Lustre: Skipped 2 previous similar messages [ 4295.668426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4295.670494] Lustre: Skipped 4 previous similar messages [ 4298.983828] LustreError: 160919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4298.991711] LustreError: 160919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4300.871923] Lustre: server umount lustre-MDT0001 complete [ 4303.552294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4308.509188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4308.623077] LustreError: 164783:0:(llog.c:1655:llog_backup()) MGC192.168.201.120@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4308.626885] Lustre: 164783:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.120@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4319.464061] LustreError: 164788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.471623] LustreError: 164788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4339.680879] LustreError: 164788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.685543] LustreError: 164788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4369.888349] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4369.892905] Lustre: 164783:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4374.759709] LustreError: 164790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.766946] LustreError: 164790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 4410.345023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4410.348235] Lustre: Skipped 2 previous similar messages [ 4411.879616] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4414.437677] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4415.263411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.008696] LustreError: 148002:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008870ebb6 x1871808111417216/t0(0) o250->MGC192.168.201.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4439.016962] LustreError: 164788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4439.018476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4439.024124] LustreError: 164788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 48 previous similar messages [ 4440.681976] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4443.581001] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4444.249879] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4444.646425] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4446.774604] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4447.492470] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4454.882260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.886070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4454.889340] Lustre: Skipped 2 previous similar messages [ 4454.893028] Lustre: Skipped 3 previous similar messages [ 4458.053304] Lustre: server umount lustre-MDT0000 complete [ 4459.554467] LustreError: 160905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785095666 with bad export cookie 8079579724305399016 [ 4459.555839] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4459.562381] LustreError: 160905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4459.707317] Lustre: server umount lustre-MDT0001 complete [ 4471.395249] Lustre: server umount lustre-OST0000 complete [ 4479.356620] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4482.866583] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4496.481460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.583201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4496.598252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4496.638223] Lustre: lustre-MDT0000: new disk, initializing [ 4496.664188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4496.666396] Lustre: Skipped 1 previous similar message [ 4496.674062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4498.391499] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4503.251639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.299871] Lustre: 169860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4503.417164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4504.976424] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4509.381904] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4511.493756] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4514.292103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4514.415109] Lustre: 171825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4516.517339] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4520.205454] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4521.465628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4521.505314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4521.638673] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4521.721162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4523.068251] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4523.146348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4526.562210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4526.562833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4526.564139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4526.564144] Lustre: Skipped 1 previous similar message [ 4526.566425] Lustre: Skipped 1 previous similar message [ 4529.951804] Lustre: server umount lustre-OST0000 complete [ 4537.561804] Lustre: server umount lustre-MDT0000 complete [ 4538.867063] LustreError: 169853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785095745 with bad export cookie 8079579724305415816 [ 4538.869511] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4538.870718] LustreError: 169853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4538.979335] Lustre: server umount lustre-MDT0001 complete [ 4541.811666] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:55:48 (1785095748) [ 4544.007037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.700549] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4548.605155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.121335] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4552.791630] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4554.727665] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4557.171250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4559.345542] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4561.858138] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4576.296136] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:56:22 (1785095782) [ 4578.785742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4578.790302] LustreError: Skipped 1 previous similar message [ 4578.792892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.798705] Lustre: Skipped 2 previous similar messages [ 4578.800723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4578.803798] Lustre: Skipped 5 previous similar messages [ 4583.452654] Lustre: server umount lustre-OST0000 complete [ 4599.264110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4599.328682] Lustre: server umount lustre-MDT0000 complete [ 4600.288933] LustreError: 174232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4600.295357] LustreError: 174232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4600.569250] LustreError: 174257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785095807 with bad export cookie 8079579724305416684 [ 4600.570990] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.573847] LustreError: 174257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4600.690995] Lustre: server umount lustre-MDT0001 complete [ 4604.225487] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 4605.241107] Key type lgssc unregistered [ 4605.377449] LNet: 177975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4605.380508] LNetError: 177975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4605.393591] LNet: Removed LNI 192.168.201.120@tcp [ 4605.693335] Key type .llcrypt unregistered [ 4605.694634] Key type ._llcrypt unregistered [ 4613.368854] Key type ._llcrypt registered [ 4613.369778] Key type .llcrypt registered [ 4613.406069] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4613.767143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4613.783147] alg: No test for adler32 (adler32-zlib) [ 4614.633812] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 4614.721743] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 4616.304138] Key type lgssc registered [ 4616.638084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4619.220557] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4619.225942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4634.656485] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4639.776395] LustreError: 179194:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 4644.896567] LustreError: 179194:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4644.900663] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4644.902485] LustreError: MGC192.168.201.120@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4644.905688] LustreError: 179194:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4644.908598] LustreError: 179194:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 4644.910731] LustreError: 179194:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4644.931839] Lustre: server umount lustre-OST0000 complete [ 4644.933322] LustreError: 179194:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4648.036539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4649.187872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4650.639343] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4653.919338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4654.063071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4655.552378] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4658.183435] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4660.138443] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4662.470592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4662.545949] LustreError: 181505:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4662.550509] Lustre: 181505:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.120@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4662.595163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.563193] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4667.177721] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4669.408753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4669.411750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.417812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4669.922413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.923202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4669.926611] Lustre: Skipped 1 previous similar message [ 4669.930522] Lustre: Skipped 1 previous similar message [ 4675.043149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4675.045143] Lustre: Skipped 1 previous similar message [ 4680.162077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4680.163925] Lustre: Skipped 2 previous similar messages [ 4682.208133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4682.274814] Lustre: server umount lustre-MDT0000 complete [ 4683.584389] LustreError: 179674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785095890 with bad export cookie 7661976189499808003 [ 4683.587714] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4683.588997] LustreError: 179674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4683.693127] Lustre: server umount lustre-MDT0001 complete [ 4695.387568] Lustre: server umount lustre-OST0000 complete [ 4698.625104] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4702.466158] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4715.459489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4715.540538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4715.550965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4715.586865] Lustre: lustre-MDT0000: new disk, initializing [ 4715.609481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.616400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4717.050907] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4721.217552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.266925] Lustre: 185022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4721.294956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4721.296418] Lustre: Skipped 1 previous similar message [ 4721.334617] Lustre: lustre-MDT0001: new disk, initializing [ 4721.370345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4721.382930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4721.387136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4722.748949] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4726.591730] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4728.534407] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4731.072610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.181592] Lustre: lustre-OST0000: new disk, initializing [ 4731.183590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4731.186131] Lustre: 186995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4731.212718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4731.286409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4731.297978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4731.329864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4733.337305] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4737.102485] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4738.499974] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4738.570886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4739.916273] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4739.986227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4741.601752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4741.602653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4741.603955] LustreError: Skipped 1 previous similar message [ 4741.604747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4741.607752] Lustre: Skipped 1 previous similar message [ 4746.699055] Lustre: server umount lustre-OST0000 complete [ 4751.841545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4751.841877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4751.846500] Lustre: Skipped 1 previous similar message [ 4751.848581] Lustre: Skipped 1 previous similar message [ 4754.141761] Lustre: server umount lustre-MDT0000 complete [ 4755.512367] LustreError: 185016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785095962 with bad export cookie 7661976189499808703 [ 4755.515089] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4755.516364] LustreError: 185016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4755.638603] Lustre: server umount lustre-MDT0001 complete [ 4758.831546] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:59:25 (1785095965) [ 4764.952873] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 4766.061517] Key type lgssc unregistered [ 4766.200491] LNet: 189936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4766.204353] LNetError: 189936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4766.213480] LNet: Removed LNI 192.168.201.120@tcp [ 4766.552640] Key type .llcrypt unregistered [ 4766.554073] Key type ._llcrypt unregistered [ 4773.937370] Key type ._llcrypt registered [ 4773.938759] Key type .llcrypt registered [ 4773.979870] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4774.397109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4774.403947] alg: No test for adler32 (adler32-zlib) [ 4775.262547] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 4775.356125] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 4776.936133] Key type lgssc registered [ 4777.316565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4780.108604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4780.114955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4781.271448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4782.667539] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4785.728807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4785.847335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4787.257155] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4789.931341] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4791.955320] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4795.060497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4795.147986] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4795.149916] LustreError: 192993:0:(llog.c:1696:llog_backup()) MGC192.168.201.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4795.153979] Lustre: 192993:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.120@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4795.200962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.481385] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4800.349558] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4801.504941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4801.508598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4801.513947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4805.602495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4805.602500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4805.609075] Lustre: Skipped 2 previous similar messages [ 4807.253174] Lustre: server umount lustre-MDT0000 complete [ 4808.490573] LustreError: 191863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785096015 with bad export cookie 11487722734882867390 [ 4808.494054] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4808.495176] LustreError: 191863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4808.500285] LustreError: 191155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4808.500465] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4808.500470] Lustre: Skipped 1 previous similar message [ 4808.501154] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4808.516978] LustreError: 191155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4808.589929] Lustre: server umount lustre-MDT0001 complete [ 4809.954393] Lustre: server umount lustre-OST0000 complete [ 4813.105175] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4816.525321] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4829.042693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.129136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4829.144150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4829.183801] Lustre: lustre-MDT0000: new disk, initializing [ 4829.206053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.211912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4830.596224] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4834.935939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.985067] Lustre: 196508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4835.008614] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4835.009775] Lustre: Skipped 1 previous similar message [ 4835.010909] LustreError: 197291:0:(llog.c:1696:llog_backup()) MGC192.168.201.120@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4835.013649] LustreError: 197291:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4835.016991] Lustre: 197291:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.120@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4835.026461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4835.028536] Lustre: Skipped 1 previous similar message [ 4835.065327] Lustre: lustre-MDT0001: new disk, initializing [ 4835.103368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4835.118644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4835.124815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4836.662643] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4840.658502] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4842.533086] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4845.012858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4845.117372] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4845.118725] Lustre: Skipped 1 previous similar message [ 4845.120395] LustreError: 198449:0:(llog.c:1696:llog_backup()) MGC192.168.201.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4845.123653] LustreError: 198449:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4845.126299] Lustre: 198449:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.120@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4845.129546] Lustre: 198449:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4845.139115] Lustre: lustre-OST0000: new disk, initializing [ 4845.141677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4845.145200] Lustre: 198473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4845.175967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4846.583715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4846.587948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4846.622752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4847.135388] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4850.759412] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4852.131254] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4852.198970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4853.513769] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4853.584035] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4856.803215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4856.804890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4856.806124] LustreError: Skipped 1 previous similar message [ 4856.806459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4856.811079] Lustre: Skipped 1 previous similar message [ 4860.307762] Lustre: server umount lustre-OST0000 complete [ 4861.921348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4861.921892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4861.925210] Lustre: Skipped 1 previous similar message [ 4861.926671] Lustre: Skipped 1 previous similar message [ 4867.674804] Lustre: server umount lustre-MDT0000 complete [ 4868.874128] LustreError: 196499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785096075 with bad export cookie 11487722734882868083 [ 4868.876806] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4868.877414] LustreError: 196499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4868.983103] Lustre: server umount lustre-MDT0001 complete [ 4871.720164] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:01:18 (1785096078) [ 4876.701667] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4879.970409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4880.115339] LustreError: 201604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4880.128391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.377675] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4884.219215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4885.615731] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4886.534626] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4888.631398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4890.650680] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4893.558734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4893.621861] Lustre: lustre-OST0001: new disk, initializing [ 4893.623861] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4893.626457] Lustre: 203910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4895.722080] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4897.090185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4897.093978] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4897.123340] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4901.073668] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4906.631361] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4907.255795] Lustre: DEBUG MARKER: ADD OST3 [ 4908.931361] loop: module loaded [ 4908.951980] loop0: detected capacity change from 0 to 204800000 [ 4909.057436] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4909.062055] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4909.066495] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4911.228516] loop0: detected capacity change from 0 to 204800000 [ 4913.189051] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.254891] Lustre: lustre-OST0003: new disk, initializing [ 4913.256969] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4913.260131] Lustre: 205638:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4913.300348] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4913.303242] Lustre: Skipped 3 previous similar messages [ 4913.993927] Lustre: DEBUG MARKER: STOP OST3 [ 4914.521638] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4914.562336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4914.567883] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4914.579105] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4914.697879] Lustre: Failing over lustre-OST0003 [ 4914.745446] Lustre: server umount lustre-OST0003 complete [ 4915.324382] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4916.576067] LustreError: 205939:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4918.921396] Lustre: DEBUG MARKER: START OST3 again [ 4919.783037] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4919.787664] LustreError: 203086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4919.787724] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4921.029058] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4921.113468] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4922.274936] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4922.800885] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4922.801298] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4922.954817] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4936.449780] Lustre: Failing over lustre-OST0003 [ 4936.485257] Lustre: server umount lustre-OST0003 complete [ 4936.678066] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4936.679650] LustreError: 203472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.683210] Lustre: Skipped 2 previous similar messages [ 4936.689068] LustreError: 203472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4941.796216] LustreError: 203086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.801787] LustreError: 203086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4951.009552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4951.012805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4951.014795] Lustre: Skipped 3 previous similar messages [ 4952.037203] LustreError: 203088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.042822] LustreError: 203088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4963.296126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4963.368248] Lustre: server umount lustre-MDT0000 complete [ 4964.544365] LustreError: 202737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785096171 with bad export cookie 11487722734882868951 [ 4964.549017] LustreError: 202737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4964.549221] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.395091] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4967.397315] Lustre: Skipped 13 previous similar messages [ 4969.442397] LustreError: 201601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.448621] LustreError: 201601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4970.738097] Lustre: server umount lustre-MDT0001 complete [ 4972.091515] Lustre: server umount lustre-OST0000 complete [ 4973.403112] Lustre: server umount lustre-OST0001 complete [ 4975.607297] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4978.287671] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4990.822797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4990.907680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4990.917331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4990.952477] Lustre: lustre-MDT0000: new disk, initializing [ 4990.974178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4990.977063] Lustre: Skipped 1 previous similar message [ 4990.982731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4992.375673] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4996.733475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.775798] Lustre: 211051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4996.887143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4998.275436] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5002.018621] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5003.807762] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5006.164551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5006.263137] Lustre: 213019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5007.672182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5007.674740] Lustre: Skipped 1 previous similar message [ 5007.677096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5007.707462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5008.247718] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5011.791854] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5013.067296] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5013.129119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5014.503889] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5014.577233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5018.081815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5018.083782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5018.084636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5018.087550] Lustre: Skipped 7 previous similar messages [ 5018.089724] Lustre: Skipped 2 previous similar messages [ 5021.270022] Lustre: server umount lustre-OST0000 complete [ 5028.698371] Lustre: server umount lustre-MDT0000 complete [ 5029.864063] LustreError: 213018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785096236 with bad export cookie 11487722734882871205 [ 5029.867561] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5029.867812] LustreError: 213018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5029.974096] Lustre: server umount lustre-MDT0001 complete [ 5031.105373] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:03:57 (1785096237) [ 5036.227596] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 5039.329414] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5050.333551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5050.429762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5050.439795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5050.441735] Lustre: Skipped 3 previous similar messages [ 5050.479695] Lustre: lustre-MDT0000: new disk, initializing [ 5050.480984] Lustre: Skipped 2 previous similar messages [ 5050.518193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5051.774693] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5055.770228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5055.813563] Lustre: 217622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5055.928553] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5057.276292] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5060.834333] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5062.599727] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5064.649753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5064.741422] Lustre: 219587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5066.139295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5066.481688] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5069.997916] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5071.420488] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5071.501157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5072.858268] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5072.924466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5076.451243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5076.453967] LustreError: Skipped 3 previous similar messages [ 5079.650220] Lustre: server umount lustre-OST0000 complete [ 5086.689067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5086.690856] Lustre: Skipped 10 previous similar messages [ 5095.392080] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5095.454176] Lustre: server umount lustre-MDT0000 complete [ 5095.905366] LustreError: 218423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.911209] LustreError: 218423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5096.684303] LustreError: 218417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785096303 with bad export cookie 11487722734882872087 [ 5096.688044] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5096.690557] LustreError: 218417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5096.799797] Lustre: server umount lustre-MDT0001 complete [ 5099.647325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5101.296509] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5104.213863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.799769] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5108.169487] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5110.044462] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5112.330314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5114.394949] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5116.956585] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5135.330367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5135.332749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5135.333610] LustreError: Skipped 1 previous similar message [ 5135.339656] Lustre: Skipped 6 previous similar messages [ 5139.918447] Lustre: server umount lustre-OST0000 complete [ 5147.283208] Lustre: server umount lustre-MDT0000 complete [ 5148.418088] LustreError: 221784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785096355 with bad export cookie 11487722734882872969 [ 5148.420969] LustreError: 221784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5148.421372] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5148.524079] Lustre: server umount lustre-MDT0001 complete [ 5250.339649] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 5251.247545] Key type lgssc unregistered [ 5251.359349] LNet: 225484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5251.361743] LNetError: 225484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5251.369240] LNet: Removed LNI 192.168.201.120@tcp [ 5251.641128] Key type .llcrypt unregistered [ 5251.642382] Key type ._llcrypt unregistered [ 5259.403856] Key type ._llcrypt registered [ 5259.404805] Key type .llcrypt registered [ 5259.441529] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 5265.351116] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5265.688930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5265.701853] alg: No test for adler32 (adler32-zlib) [ 5266.535843] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 5266.611554] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 5268.184141] Key type lgssc registered [ 5268.574551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5279.879730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5279.885992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5280.968669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5280.978477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5281.013917] Lustre: lustre-MDT0000: new disk, initializing [ 5281.033049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.038624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5282.388055] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5286.593813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5286.631584] Lustre: 228808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5286.654211] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5286.656551] Lustre: Skipped 1 previous similar message [ 5286.697022] Lustre: lustre-MDT0001: new disk, initializing [ 5286.728644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5286.738595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5286.741651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5288.073321] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5291.642323] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5293.514626] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5295.834436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5295.956091] Lustre: lustre-OST0000: new disk, initializing [ 5295.958471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5295.962609] Lustre: 230776:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5295.991968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5297.948532] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5301.562672] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5302.779405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5302.783289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5302.820444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5302.926516] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5302.990396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5304.300852] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5304.376827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5307.873385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5307.875848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.876650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5307.880306] Lustre: Skipped 1 previous similar message [ 5311.124625] Lustre: server umount lustre-OST0000 complete [ 5312.480890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5312.482904] LustreError: Skipped 1 previous similar message [ 5312.484303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5312.489153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5312.491214] Lustre: Skipped 1 previous similar message [ 5318.113129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5318.115747] Lustre: Skipped 2 previous similar messages [ 5318.427813] Lustre: server umount lustre-MDT0000 complete [ 5319.576993] LustreError: 229603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785096526 with bad export cookie 11823572532953869346 [ 5319.578689] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5319.580970] LustreError: 229603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5319.694021] Lustre: server umount lustre-MDT0001 complete [ 5325.526927] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5328.960257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.111678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5330.460185] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5333.202942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.327443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5334.702757] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5335.697139] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5337.929349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5340.002563] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5342.873095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5342.945288] Lustre: lustre-OST0001: new disk, initializing [ 5342.947220] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5342.950402] Lustre: 236009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5342.989349] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5342.991814] Lustre: Skipped 1 previous similar message [ 5344.945205] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5345.985545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5345.988496] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5346.011383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5350.353533] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5354.941348] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5358.519342] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:09:25 (1785096565) [ 5361.634096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5361.636578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5361.639703] Lustre: Skipped 4 previous similar messages [ 5361.642421] Lustre: Skipped 3 previous similar messages [ 5365.335956] Lustre: server umount lustre-MDT0000 complete [ 5366.547711] LustreError: 233686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785096573 with bad export cookie 11823572532953870214 [ 5366.553555] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5366.667790] Lustre: server umount lustre-MDT0001 complete [ 5378.117189] Lustre: server umount lustre-OST0000 complete [ 5389.594423] Lustre: server umount lustre-OST0001 complete [ 5391.705800] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 5395.279959] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5407.849350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.952746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5407.964571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5408.007652] Lustre: lustre-MDT0000: new disk, initializing [ 5408.031776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5408.038291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5409.452176] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5413.799289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5413.843873] Lustre: 239950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5413.963701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5415.475761] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5419.266399] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5421.164873] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5423.433727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5423.539957] Lustre: lustre-OST0000: new disk, initializing [ 5423.541773] Lustre: Skipped 1 previous similar message [ 5423.544215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5423.546666] Lustre: Skipped 2 previous similar messages [ 5423.548696] Lustre: 241918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5425.339591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5425.343599] Lustre: Skipped 1 previous similar message [ 5425.346210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5425.385500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5425.538353] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5429.335746] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5430.893335] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5430.980550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5432.423576] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5432.503309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5435.874316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5435.876589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5435.877449] LustreError: Skipped 1 previous similar message [ 5435.883400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5435.887376] Lustre: Skipped 1 previous similar message [ 5439.250628] Lustre: server umount lustre-OST0000 complete [ 5440.993419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5440.999691] Lustre: Skipped 2 previous similar messages [ 5446.113434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5446.115950] Lustre: Skipped 3 previous similar messages [ 5446.682485] Lustre: server umount lustre-MDT0000 complete [ 5447.903327] LustreError: 239945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785096655 with bad export cookie 11823572532953872307 [ 5447.905717] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5447.907969] LustreError: 239945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5448.037704] Lustre: server umount lustre-MDT0001 complete [ 5450.880870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.042994] LustreError: 244133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5451.048138] LustreError: 244133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5451.070513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5451.073143] Lustre: Skipped 2 previous similar messages [ 5452.410884] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5455.267042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5456.777916] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5459.218645] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5461.092774] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5463.328989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5465.351104] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5467.892215] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5614.049666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5614.053054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5614.058022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5618.710485] Lustre: server umount lustre-OST0000 complete [ 5626.203546] Lustre: server umount lustre-MDT0000 complete [ 5627.550726] LustreError: 244115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785096834 with bad export cookie 11823572532953873189 [ 5627.553542] LustreError: 244115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5627.554649] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5627.686864] Lustre: server umount lustre-MDT0001 complete [ 5730.096966] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 5731.042324] Key type lgssc unregistered [ 5731.176354] LNet: 247737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5731.179744] LNetError: 247737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5731.188269] LNet: Removed LNI 192.168.201.120@tcp [ 5731.495099] Key type .llcrypt unregistered [ 5731.496079] Key type ._llcrypt unregistered [ 5739.556366] Key type ._llcrypt registered [ 5739.557434] Key type .llcrypt registered [ 5739.597429] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 5745.779836] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5746.145383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5746.186258] alg: No test for adler32 (adler32-zlib) [ 5747.026594] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 5747.103729] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 5748.680189] Key type lgssc registered [ 5749.019128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5762.743804] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5765.813631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5765.818885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.899502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5766.910951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5766.946530] Lustre: lustre-MDT0000: new disk, initializing [ 5766.965270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5766.970301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5768.218598] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5772.142499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5772.176326] Lustre: 251779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5772.196091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5772.197902] Lustre: Skipped 1 previous similar message [ 5772.233912] Lustre: lustre-MDT0001: new disk, initializing [ 5772.272122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5772.287459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5772.291421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5773.702217] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5775.970930] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5778.155900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5778.270100] Lustre: lustre-OST0000: new disk, initializing [ 5778.271679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5778.274112] Lustre: 253409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5778.306624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5780.190909] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5782.523828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5782.527220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5782.565938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5784.255972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5784.313954] Lustre: lustre-OST0001: new disk, initializing [ 5784.316091] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5784.318328] Lustre: 254283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5784.351140] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5786.207761] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5790.732091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5790.736924] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5790.763596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5791.692687] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5797.485225] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5800.930534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5800.932684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5800.934230] Lustre: Skipped 3 previous similar messages [ 5800.935857] Lustre: Skipped 2 previous similar messages [ 5806.049486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5806.052386] Lustre: Skipped 3 previous similar messages [ 5806.696055] Lustre: server umount lustre-MDT0000 complete [ 5807.965224] LustreError: 251769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785097015 with bad export cookie 16729400388117637705 [ 5807.969084] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5807.970057] LustreError: 251769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5808.097400] Lustre: server umount lustre-MDT0001 complete [ 5819.399221] Lustre: server umount lustre-OST0000 complete [ 5830.943237] Lustre: server umount lustre-OST0001 complete [ 5835.698638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5838.932229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5841.808499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5845.039718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5851.657629] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5855.236623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.243395] Lustre: lustre-MDT0000: reset Object Index mappings [ 5855.419648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5856.843835] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5859.815107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5859.822186] Lustre: lustre-MDT0001: reset Object Index mappings [ 5861.420881] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5862.482634] Lustre: 260426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5864.813262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5864.948854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5864.952290] Lustre: Skipped 1 previous similar message [ 5865.957510] LustreError: 260779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.961286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5865.966364] LustreError: 260779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5867.088060] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5870.053908] LustreError: 260779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.060488] LustreError: 260779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5870.069564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5872.149488] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5875.181689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5875.312693] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5876.497610] Lustre: 262292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5880.292598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.293945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5880.297093] Lustre: Skipped 2 previous similar messages [ 5880.299661] Lustre: Skipped 1 previous similar message [ 5883.811207] Lustre: server umount lustre-MDT0000 complete [ 5885.087823] LustreError: 260028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785097092 with bad export cookie 16729400388117644481 [ 5885.091098] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5885.092593] LustreError: 260028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5885.212990] Lustre: server umount lustre-MDT0001 complete [ 5896.775331] Lustre: server umount lustre-OST0000 complete [ 5908.257580] Lustre: server umount lustre-OST0001 complete [ 5914.705300] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:18:41 (1785097121) [ 5920.078510] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 5923.725635] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5936.224316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5936.302059] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5936.312140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5936.349311] Lustre: lustre-MDT0000: new disk, initializing [ 5936.371412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.373627] Lustre: Skipped 1 previous similar message [ 5936.378528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5937.778455] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5941.823414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5941.868850] Lustre: 266536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5941.977472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5943.287862] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5946.704411] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5948.524408] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5950.812656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5950.927592] Lustre: lustre-OST0000: new disk, initializing [ 5950.928793] Lustre: Skipped 1 previous similar message [ 5950.930211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5950.932143] Lustre: Skipped 2 previous similar messages [ 5950.933946] Lustre: 268501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5952.635880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5952.638241] Lustre: Skipped 1 previous similar message [ 5952.639918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5952.678108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5952.995844] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5956.797185] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5958.214492] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5958.289256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5959.628528] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5959.697278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5963.236124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5963.236381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.241983] Lustre: Skipped 1 previous similar message [ 5963.244157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.246247] Lustre: Skipped 3 previous similar messages [ 5966.416554] Lustre: server umount lustre-OST0000 complete [ 5967.840903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5967.843823] LustreError: Skipped 1 previous similar message [ 5967.846338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.850106] Lustre: Skipped 1 previous similar message [ 5967.852372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5973.790126] Lustre: server umount lustre-MDT0000 complete [ 5975.033636] LustreError: 266529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785097182 with bad export cookie 16729400388117646553 [ 5975.036268] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5975.036992] LustreError: 266529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5975.163202] Lustre: server umount lustre-MDT0001 complete [ 5981.609614] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5985.063263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5985.186062] LustreError: 271429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5985.197504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5985.199789] Lustre: Skipped 2 previous similar messages [ 5986.668610] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5989.565255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.213702] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5992.225886] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5994.535690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5996.787933] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5999.712397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5999.801808] Lustre: lustre-OST0001: new disk, initializing [ 5999.803996] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5999.808742] Lustre: 273734:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6000.842867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6000.846091] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6000.888562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6001.997421] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6006.556463] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6007.801412] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6009.684027] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6009.689844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6015.457464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6015.461439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6015.465723] Lustre: Skipped 1 previous similar message [ 6015.467300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6015.469544] Lustre: Skipped 4 previous similar messages [ 6017.570782] Lustre: server umount lustre-MDT0000 complete [ 6018.865491] LustreError: 271412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785097225 with bad export cookie 16729400388117647435 [ 6018.869771] LustreError: 271412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6018.870205] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6018.995374] Lustre: server umount lustre-MDT0001 complete [ 6030.853457] Lustre: server umount lustre-OST0000 complete [ 6042.405993] Lustre: server umount lustre-OST0001 complete [ 6048.106987] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6051.589299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.729965] LustreError: 276786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.746718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.750058] Lustre: Skipped 3 previous similar messages [ 6053.198315] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6056.165545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.752082] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6058.770727] Lustre: 277926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6060.994662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6062.181221] LustreError: 278280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6062.187220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6063.104814] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6065.994173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6067.116905] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6068.065881] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6071.270261] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6072.518054] Lustre: 279800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6074.397670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6074.403442] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6077.471284] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:21:24 (1785097284) [ 6092.257203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6092.259544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6092.263290] Lustre: Skipped 3 previous similar messages [ 6092.265264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6092.267994] Lustre: Skipped 4 previous similar messages [ 6095.204913] Lustre: server umount lustre-MDT0000 complete [ 6096.451505] LustreError: 276768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785097303 with bad export cookie 16729400388117653735 [ 6096.455848] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6096.456696] LustreError: 276768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6097.889172] LustreError: 276815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6097.893789] LustreError: 276815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6102.639395] Lustre: server umount lustre-MDT0001 complete [ 6118.368113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6118.409545] Lustre: server umount lustre-OST0000 complete [ 6125.789709] Lustre: server umount lustre-OST0001 complete [ 6127.899459] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 6131.218529] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6146.365560] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6149.994750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6150.097089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6150.106878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6150.145721] Lustre: lustre-MDT0000: new disk, initializing [ 6150.170887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6151.583867] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6155.703580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6155.746933] Lustre: 284033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6155.864062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6157.244671] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6161.728130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6161.849275] Lustre: 285663:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6163.894280] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6167.039422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6168.143355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6169.705900] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6170.190803] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6174.724544] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6180.229421] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:23:06 (1785097386) [ 6181.345053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6181.347344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6181.350519] Lustre: Skipped 5 previous similar messages [ 6181.352241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6181.354760] Lustre: Skipped 7 previous similar messages [ 6187.107645] Lustre: server umount lustre-MDT0000 complete [ 6188.362201] LustreError: 284025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785097395 with bad export cookie 16729400388117661617 [ 6188.366724] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.366916] LustreError: 284025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6188.492584] Lustre: server umount lustre-MDT0001 complete [ 6199.753381] Lustre: server umount lustre-OST0000 complete [ 6211.289900] Lustre: server umount lustre-OST0001 complete [ 6214.149591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.299881] LustreError: 288774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6214.305655] LustreError: 288774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6214.337773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6214.339892] Lustre: Skipped 7 previous similar messages [ 6215.748770] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6218.629823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6220.164468] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6222.629415] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6224.502867] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6226.770521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6228.908431] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6231.327854] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6247.144719] LustreError: 290601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.150230] LustreError: 290601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6248.000895] loop1: detected capacity change from 0 to 204800000 [ 6248.143400] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6248.148567] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6248.153865] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6249.919893] loop1: detected capacity change from 0 to 204800000 [ 6251.854228] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6251.888335] LustreError: 288763:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs18344: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6251.892654] LustreError: 288763:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs18344 [ 6251.894722] LustreError: 288763:0:(mgs_handler.c:517:mgs_target_reg()) fs18344-MDT0001: Error checking target index: rc = -13 [ 6251.897968] LustreError: fs18344-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6251.900902] LustreError: 292510:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6251.903256] LustreError: 292510:0:(tgt_mount.c:2129:server_put_super()) no obd fs18344-MDT0001 [ 6251.905171] LustreError: 292510:0:(tgt_mount.c:116:server_deregister_mount()) fs18344-MDT0001 not registered [ 6251.916246] Lustre: server umount fs18344-MDT0001 complete [ 6251.917808] LustreError: 292510:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6254.832183] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.861255] Lustre: Setting parameter fs18344-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs18344-MDT0001 [ 6254.865034] Lustre: Skipped 2 previous similar messages [ 6254.880257] Lustre: srv-fs18344-MDT0001: No data found on store. Initialize space. [ 6254.882246] Lustre: Skipped 4 previous similar messages [ 6254.922388] Lustre: fs18344-MDT0001: new disk, initializing [ 6254.924579] Lustre: Skipped 3 previous similar messages [ 6254.959166] Lustre: srv-fs18344-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6254.961467] Lustre: 292975:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18344-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6254.963937] LustreError: 292975:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18344-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6256.339732] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6259.257115] Lustre: server umount fs18344-MDT0001 complete [ 6263.601388] loop1: detected capacity change from 0 to 204800000 [ 6265.467166] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.593327] Lustre: 294541:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18344-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6265.596149] LustreError: 294541:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18344-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6266.926843] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6268.545917] Lustre: server umount fs18344-MDT0001 complete [ 6272.750326] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 6272.752270] Lustre: Skipped 4 previous similar messages [ 6275.044639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6275.044919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6275.050243] Lustre: Skipped 4 previous similar messages [ 6277.869377] Lustre: server umount lustre-MDT0000 complete [ 6280.879708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6280.928783] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6282.519884] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6285.173857] loop1: detected capacity change from 0 to 204800000 [ 6286.313447] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6287.173871] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6287.305542] Lustre: 296984:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18344-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6287.310450] LustreError: 296984:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18344-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6288.687647] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6290.292537] Lustre: server umount fs18344-MDT0001 complete [ 6299.616933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6311.392146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6311.444689] Lustre: server umount lustre-OST0000 complete [ 6318.811379] Lustre: server umount lustre-MDT0000 complete [ 6320.046347] LustreError: 288756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785097527 with bad export cookie 16729400388117664291 [ 6320.049264] LustreError: 288756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6320.050535] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6320.166163] Lustre: server umount lustre-MDT0001 complete [ 6323.931434] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6325.003112] Key type lgssc unregistered [ 6325.128457] LNet: 299322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6325.131738] LNetError: 299322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6325.143389] LNet: Removed LNI 192.168.201.120@tcp [ 6325.485115] Key type .llcrypt unregistered [ 6325.486243] Key type ._llcrypt unregistered [ 6333.406439] Key type ._llcrypt registered [ 6333.407599] Key type .llcrypt registered [ 6333.447368] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6333.822495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6333.877963] alg: No test for adler32 (adler32-zlib) [ 6334.730205] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6334.831402] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6336.424137] Key type lgssc registered [ 6336.819761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6339.566132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6339.571248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.739748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.173388] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6345.072078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6345.225135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6346.689201] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6349.128575] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6351.087475] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6353.389837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6353.534458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6355.489801] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6355.622757] LustreError: 302400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.627171] LustreError: 302400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6357.978205] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6361.065281] LustreError: 302400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.185877] LustreError: 302400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.190209] LustreError: 302400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6368.496458] LustreError: 302785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6373.610596] LustreError: 302400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6373.616102] LustreError: 302400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6374.544321] loop1: detected capacity change from 0 to 204800000 [ 6374.604055] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6374.610700] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6374.616053] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6376.355936] loop1: detected capacity change from 0 to 204800000 [ 6378.221987] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6378.273205] Lustre: fs27745-OST0000: new disk, initializing [ 6378.274771] Lustre: srv-fs27745-OST0000: No data found on store. Initialize space. [ 6378.277137] Lustre: 304327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6378.297944] Lustre: fs27745-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6380.318855] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6383.849463] LustreError: 302400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.856485] LustreError: 302400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6383.862242] Lustre: server umount fs27745-OST0000 complete [ 6388.157520] loop1: detected capacity change from 0 to 204800000 [ 6390.193726] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6390.268881] Lustre: fs27745-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6392.340108] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6395.346442] loop2: detected capacity change from 0 to 204800000 [ 6395.431940] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6395.435583] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6395.439413] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6397.267069] loop2: detected capacity change from 0 to 204800000 [ 6399.282737] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.322057] LustreError: fs27745-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6399.327160] LustreError: fs27745-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6399.331911] LustreError: 307363:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6399.335234] LustreError: 307363:0:(tgt_mount.c:2129:server_put_super()) no obd fs27745-OST0001 [ 6399.338108] LustreError: 307363:0:(tgt_mount.c:116:server_deregister_mount()) fs27745-OST0001 not registered [ 6399.350650] Lustre: server umount fs27745-OST0001 complete [ 6399.352709] LustreError: 307363:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6402.021124] LustreError: 302401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6402.025689] LustreError: 302401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6402.773672] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6402.825284] Lustre: fs27745-OST0001: new disk, initializing [ 6402.827909] Lustre: srv-fs27745-OST0001: No data found on store. Initialize space. [ 6402.831836] Lustre: 307914:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6402.856871] Lustre: fs27745-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6404.924453] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6408.410201] Lustre: server umount fs27745-OST0000 complete [ 6411.553358] Lustre: server umount fs27745-OST0001 complete [ 6415.648515] loop1: detected capacity change from 0 to 204800000 [ 6417.591342] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6419.705364] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6421.997570] loop2: detected capacity change from 0 to 204800000 [ 6423.892524] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6423.959031] Lustre: fs27745-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6423.961295] Lustre: Skipped 1 previous similar message [ 6425.979495] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6429.049452] Lustre: server umount fs27745-OST0000 complete [ 6432.231168] Lustre: server umount fs27745-OST0001 complete [ 6435.047930] LustreError: 302401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6435.055293] LustreError: 302401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 6437.859889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6437.860971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6437.864126] Lustre: Skipped 1 previous similar message [ 6437.864475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.866355] LustreError: Skipped 1 previous similar message [ 6437.867606] Lustre: Skipped 1 previous similar message [ 6442.065979] Lustre: server umount lustre-OST0000 complete [ 6447.584948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6447.586900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6447.590447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6449.440762] Lustre: server umount lustre-MDT0000 complete [ 6450.714175] LustreError: 300551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785097657 with bad export cookie 1752853369082229989 [ 6450.716695] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6450.718105] LustreError: 300551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6450.841392] Lustre: server umount lustre-MDT0001 complete [ 6454.643688] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6455.770902] Key type lgssc unregistered [ 6455.897373] LNet: 313822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6455.900142] LNetError: 313822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6455.909273] LNet: Removed LNI 192.168.201.120@tcp [ 6456.268145] Key type .llcrypt unregistered [ 6456.269697] Key type ._llcrypt unregistered [ 6467.011186] Key type ._llcrypt registered [ 6467.012445] Key type .llcrypt registered [ 6467.057519] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 6473.351594] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6473.731895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6473.772835] alg: No test for adler32 (adler32-zlib) [ 6474.638039] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6474.732176] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6476.312160] Key type lgssc registered [ 6476.669930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6488.628709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6488.635761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.715933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6489.725863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6489.763284] Lustre: lustre-MDT0000: new disk, initializing [ 6489.784753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6489.790896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6491.129593] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6495.274229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6495.313782] Lustre: 317651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6495.335096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6495.337177] Lustre: Skipped 1 previous similar message [ 6495.373344] Lustre: lustre-MDT0001: new disk, initializing [ 6495.404898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6495.417051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6495.420749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6496.853553] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6500.587305] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6502.444716] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6504.658825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6504.760060] Lustre: lustre-OST0000: new disk, initializing [ 6504.761817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6504.764966] Lustre: 319620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6504.793922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6505.851964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6505.854756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6505.882072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6506.693916] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6510.298971] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6511.050313] Lustre: Mounted lustre-client [ 6511.428192] Lustre: Unmounted lustre-client [ 6516.193735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6516.196346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.197421] LustreError: Skipped 1 previous similar message [ 6516.197641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6516.200960] Lustre: Skipped 1 previous similar message [ 6520.597630] Lustre: server umount lustre-OST0000 complete [ 6526.433934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.434050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6526.434560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.434564] Lustre: Skipped 1 previous similar message [ 6526.437476] Lustre: Skipped 1 previous similar message [ 6528.086170] Lustre: server umount lustre-MDT0000 complete [ 6529.302170] LustreError: 317645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785097736 with bad export cookie 8765773694129166624 [ 6529.305165] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6529.305724] LustreError: 317645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6529.420737] Lustre: server umount lustre-MDT0001 complete [ 6533.140834] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6534.126036] Key type lgssc unregistered [ 6567.008360] LNet: 321779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6567.010713] LNetError: 321779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6640.742564] LNet: Removed LNI 192.168.201.120@tcp [ 6641.059611] Key type .llcrypt unregistered [ 6641.060706] Key type ._llcrypt unregistered [ 6646.901979] Key type ._llcrypt registered [ 6646.903673] Key type .llcrypt registered [ 6646.940491] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 6653.350275] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6653.661609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6653.670735] alg: No test for adler32 (adler32-zlib) [ 6654.521696] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6654.610113] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6656.184115] Key type lgssc registered [ 6656.531586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6666.053870] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:31:12 (1785097872) [ 6668.353768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6668.359023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6669.440259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6669.453562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6669.489793] Lustre: lustre-MDT0000: new disk, initializing [ 6669.511372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.517571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6670.876469] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6675.494710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.539516] Lustre: 324976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6675.564760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6675.567864] Lustre: Skipped 1 previous similar message [ 6675.612628] Lustre: lustre-MDT0001: new disk, initializing [ 6675.644631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6675.655957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6675.659601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6676.975553] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6680.672332] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6682.500645] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6684.672130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6684.769114] Lustre: lustre-OST0000: new disk, initializing [ 6684.770799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6684.773369] Lustre: 326943:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6684.801261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6685.753636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6685.757657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6685.789786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6686.578067] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6690.159790] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6695.906057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6695.908445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.910805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6695.912795] Lustre: Skipped 1 previous similar message [ 6697.624997] Lustre: server umount lustre-OST0000 complete [ 6701.026763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6701.028509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6701.031915] Lustre: Skipped 1 previous similar message [ 6701.033493] Lustre: Skipped 1 previous similar message [ 6705.044036] Lustre: server umount lustre-MDT0000 complete [ 6706.313565] LustreError: 324968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785097913 with bad export cookie 3779790320573618110 [ 6706.318903] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6706.451941] Lustre: server umount lustre-MDT0001 complete [ 6710.023395] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6710.993860] Key type lgssc unregistered [ 6711.112478] LNet: 328847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6711.115917] LNetError: 328847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6711.125303] LNet: Removed LNI 192.168.201.120@tcp [ 6711.454312] Key type .llcrypt unregistered [ 6711.455648] Key type ._llcrypt unregistered [ 6719.796405] Key type ._llcrypt registered [ 6719.797473] Key type .llcrypt registered [ 6719.832035] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6720.243485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6720.277212] alg: No test for adler32 (adler32-zlib) [ 6721.126262] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6721.215653] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6722.792141] Key type lgssc registered [ 6723.154583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6725.660179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6725.667031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.812052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6728.139509] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6730.911063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.051375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6732.415962] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6734.696613] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6736.522428] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6738.724559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6738.835581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6740.710141] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6743.082930] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6757.167874] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6757.630248] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6761.442761] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.444716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6761.446026] Lustre: Skipped 2 previous similar messages [ 6761.448868] Lustre: Skipped 1 previous similar message [ 6763.759568] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 6766.562706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6766.564752] Lustre: Skipped 1 previous similar message [ 6768.872525] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 6773.216151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6773.281767] Lustre: server umount lustre-MDT0000 complete [ 6773.992108] LustreError: 330092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6773.997797] LustreError: 330092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6774.561368] LustreError: 330077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785097981 with bad export cookie 7804864034945307440 [ 6774.564943] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6774.565162] LustreError: 330077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6776.800946] LustreError: 330093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6776.801353] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.812989] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6776.815116] Lustre: Skipped 3 previous similar messages [ 6779.111759] LustreError: 330091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6780.714097] Lustre: server umount lustre-MDT0001 complete [ 6782.045500] Lustre: server umount lustre-OST0000 complete [ 6785.013043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6786.348590] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6787.337642] Lustre: 334294:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.120@tcp [ 6788.059254] Lustre: server umount MGS complete [ 6791.184507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6791.353494] LustreError: 334865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6791.358333] LustreError: 334865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6791.373812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6792.727749] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6796.207847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.378135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6797.935989] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6801.259049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6803.722980] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6806.437095] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6807.149677] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 6809.149461] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6810.698633] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 6812.640939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6812.643130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6812.647071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6812.648716] Lustre: Skipped 2 previous similar messages [ 6816.742514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6816.745028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.873244] Lustre: server umount lustre-OST0000 complete [ 6821.859339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6821.859930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6821.867349] Lustre: Skipped 1 previous similar message [ 6825.247756] Lustre: server umount lustre-MDT0000 complete [ 6826.518583] LustreError: 336321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785098033 with bad export cookie 7804864034945308266 [ 6826.523524] LustreError: 336321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6826.523821] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6826.656375] Lustre: server umount lustre-MDT0001 complete [ 6830.805488] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6831.937443] Key type lgssc unregistered [ 6832.068313] LNet: 338298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6832.070596] LNetError: 338298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6832.076281] LNet: Removed LNI 192.168.201.120@tcp [ 6832.421310] Key type .llcrypt unregistered [ 6832.423102] Key type ._llcrypt unregistered [ 6838.570296] Key type ._llcrypt registered [ 6838.571270] Key type .llcrypt registered [ 6838.606814] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 6844.826615] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6845.169119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6845.234196] alg: No test for adler32 (adler32-zlib) [ 6846.084361] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6846.166917] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6847.744179] Key type lgssc registered [ 6848.133909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6860.006486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6860.013042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6861.103930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6861.114122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6861.148757] Lustre: lustre-MDT0000: new disk, initializing [ 6861.168670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.176522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6862.507786] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6866.762647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.803428] Lustre: 341428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6866.827891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6866.829912] Lustre: Skipped 1 previous similar message [ 6866.869146] Lustre: lustre-MDT0001: new disk, initializing [ 6866.902426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6866.915278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6866.918227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6868.275599] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6872.301257] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6874.560703] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6876.867066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6876.968682] Lustre: lustre-OST0000: new disk, initializing [ 6876.970244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6876.972516] Lustre: 343396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6877.001514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6878.921395] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6882.651915] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6882.810401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6882.813362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6882.857067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6884.231136] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6884.329744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6885.666080] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6885.730099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6887.906919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6887.907778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6887.909659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6887.914300] Lustre: Skipped 1 previous similar message [ 6892.439559] Lustre: server umount lustre-OST0000 complete [ 6897.632897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6897.636319] LustreError: Skipped 1 previous similar message [ 6897.638331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6897.644338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6897.646110] Lustre: Skipped 1 previous similar message [ 6903.265601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6903.268687] Lustre: Skipped 3 previous similar messages [ 6907.872108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6907.943612] Lustre: server umount lustre-MDT0000 complete [ 6909.075172] LustreError: 343395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785098116 with bad export cookie 8420098158319553532 [ 6909.078348] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6909.079777] LustreError: 343395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6909.188743] Lustre: server umount lustre-MDT0001 complete [ 6912.075883] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:35:18 (1785098118) [ 6917.000327] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 6920.367253] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6932.431749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.512886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6932.522755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6932.557879] Lustre: lustre-MDT0000: new disk, initializing [ 6932.576713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6932.584402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6933.942951] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6938.104389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6938.147733] Lustre: 348144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6938.170486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6938.172189] Lustre: Skipped 1 previous similar message [ 6938.206462] Lustre: lustre-MDT0001: new disk, initializing [ 6938.237302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6938.248180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6938.251069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6939.568243] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6943.169544] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6944.937928] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6947.249119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6947.349689] Lustre: lustre-OST0000: new disk, initializing [ 6947.351364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6947.353617] Lustre: 350113:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6947.378981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6949.292958] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6949.624917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6949.628294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6949.667669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6953.324636] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6965.220228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6965.221287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.223646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6965.226926] Lustre: Skipped 2 previous similar messages [ 6965.230873] Lustre: Skipped 1 previous similar message [ 6969.232481] Lustre: server umount lustre-OST0000 complete [ 6974.433038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6974.436366] LustreError: Skipped 1 previous similar message [ 6974.438599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.445748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6976.677182] Lustre: server umount lustre-MDT0000 complete [ 6977.851277] LustreError: 348136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785098184 with bad export cookie 8420098158319554400 [ 6977.855910] LustreError: 348136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6977.856502] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6977.966334] Lustre: server umount lustre-MDT0001 complete [ 6981.736475] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6982.766252] Key type lgssc unregistered [ 6982.888356] LNet: 352261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6982.890670] LNetError: 352261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6982.898288] LNet: Removed LNI 192.168.201.120@tcp [ 6983.208957] Key type .llcrypt unregistered [ 6983.210236] Key type ._llcrypt unregistered [ 7015.048139] Key type ._llcrypt registered [ 7015.049224] Key type .llcrypt registered [ 7015.092699] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7015.206412] Key type .llcrypt unregistered [ 7015.207885] Key type ._llcrypt unregistered [ 7043.509250] Key type ._llcrypt registered [ 7043.510306] Key type .llcrypt registered [ 7043.541670] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7043.635168] Key type .llcrypt unregistered [ 7043.636188] Key type ._llcrypt unregistered [ 7054.382511] Key type ._llcrypt registered [ 7054.383495] Key type .llcrypt registered [ 7054.425615] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7054.524226] Key type .llcrypt unregistered [ 7054.525238] Key type ._llcrypt unregistered [ 7064.882161] Key type ._llcrypt registered [ 7064.883127] Key type .llcrypt registered [ 7064.915199] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7065.001440] Key type .llcrypt unregistered [ 7065.003135] Key type ._llcrypt unregistered [ 7090.984403] Key type ._llcrypt registered [ 7090.985355] Key type .llcrypt registered [ 7091.017481] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7091.104195] Key type .llcrypt unregistered [ 7091.105283] Key type ._llcrypt unregistered [ 7101.888687] Key type ._llcrypt registered [ 7101.889876] Key type .llcrypt registered [ 7101.925085] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7102.025299] Key type .llcrypt unregistered [ 7102.026895] Key type ._llcrypt unregistered [ 7112.707700] Key type ._llcrypt registered [ 7112.709323] Key type .llcrypt registered [ 7112.750348] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7112.847276] Key type .llcrypt unregistered [ 7112.848368] Key type ._llcrypt unregistered [ 7123.591876] Key type ._llcrypt registered [ 7123.593010] Key type .llcrypt registered [ 7123.633718] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7123.742273] Key type .llcrypt unregistered [ 7123.744088] Key type ._llcrypt unregistered [ 7134.467254] Key type ._llcrypt registered [ 7134.468271] Key type .llcrypt registered [ 7134.502844] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7134.594354] Key type .llcrypt unregistered [ 7134.595373] Key type ._llcrypt unregistered [ 7145.657410] Key type ._llcrypt registered [ 7145.658395] Key type .llcrypt registered [ 7145.692100] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7145.782248] Key type .llcrypt unregistered [ 7145.783458] Key type ._llcrypt unregistered [ 7166.458152] Key type ._llcrypt registered [ 7166.459250] Key type .llcrypt registered [ 7166.493954] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7166.590272] Key type .llcrypt unregistered [ 7166.591343] Key type ._llcrypt unregistered [ 7177.237773] Key type ._llcrypt registered [ 7177.239522] Key type .llcrypt registered [ 7177.276689] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7177.363195] Key type .llcrypt unregistered [ 7177.364182] Key type ._llcrypt unregistered [ 7188.017563] Key type ._llcrypt registered [ 7188.018623] Key type .llcrypt registered [ 7188.053677] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7188.143321] Key type .llcrypt unregistered [ 7188.144520] Key type ._llcrypt unregistered [ 7205.483330] Key type ._llcrypt registered [ 7205.484508] Key type .llcrypt registered [ 7205.521868] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 7211.990721] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7212.286025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7212.293598] alg: No test for adler32 (adler32-zlib) [ 7213.143062] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7213.228697] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7214.808148] Key type lgssc registered [ 7215.320361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7227.854799] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7227.860205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.937164] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7228.945934] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7228.985489] Lustre: tst250fs-MDT0000: new disk, initializing [ 7229.005130] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7229.010108] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7230.244449] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7235.059550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7235.131282] Lustre: 372161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7235.176422] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7235.180445] Lustre: Skipped 1 previous similar message [ 7235.232444] Lustre: tst250fs-MDT0001: new disk, initializing [ 7235.269876] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7235.286692] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7235.290882] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7236.913279] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7239.778772] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7242.945381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7243.084541] Lustre: tst250fs-OST0000: new disk, initializing [ 7243.086174] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7243.088758] Lustre: 373789:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7243.117862] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7244.439473] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7244.445227] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7244.514194] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7245.645816] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7250.277234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.353637] Lustre: tst250fs-OST0001: new disk, initializing [ 7250.355326] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7250.357847] Lustre: 374662:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7250.390430] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7251.295757] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7251.300934] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7251.337580] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7252.812828] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7255.791545] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7271.392980] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7271.396120] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.401621] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.907159] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.909730] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.912231] Lustre: Skipped 2 previous similar messages [ 7271.916638] Lustre: Skipped 2 previous similar messages [ 7273.710725] Lustre: server umount tst250fs-MDT0000 complete [ 7275.293123] LustreError: 374661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785098482 with bad export cookie 17834240921165169404 [ 7275.299249] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7275.300542] LustreError: 374661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7275.441651] Lustre: server umount tst250fs-MDT0001 complete [ 7287.492419] Lustre: server umount tst250fs-OST0000 complete [ 7299.453800] Lustre: server umount tst250fs-OST0001 complete [ 7307.344515] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 7311.163297] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7324.743792] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:42:11 (1785098531) [ 7325.548561] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7331.487615] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 7336.383966] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7349.194516] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7101 sec ========== 16:42:35 (1785098555) [ 7349.798862] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:42:36 (1785098556) === [ 7351.108600] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:42:37 (1785098557) === [ 7366.141566] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7367.090418] Key type lgssc unregistered [ 7367.209361] LNet: 383777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7367.212506] LNetError: 383777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7367.219285] LNet: Removed LNI 192.168.201.120@tcp [ 7367.510097] Key type .llcrypt unregistered [ 7367.511256] Key type ._llcrypt unregistered