[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 456856869 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002381] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009127] LSM: Security Framework initializing [ 0.010043] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.012074] *** VALIDATE selinux *** [ 0.020332] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025544] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029117] *** VALIDATE tmpfs *** [ 0.031460] *** VALIDATE proc *** [ 0.033048] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.041950] debug: unmapping init [mem 0xffffffff92a59000-0xffffffff92a60fff] [ 0.043171] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044623] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052296] rcu: Hierarchical SRCU implementation. [ 0.054382] smp: Bringing up secondary CPUs ... [ 0.055594] x86: Booting SMP configuration: [ 0.056028] .... node #0, CPUs: #1 #2 #3 [ 0.059467] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138098] node 0 deferred pages initialised in 74ms [ 0.142038] devtmpfs: initialized [ 0.143225] x86/mm: Memory block size: 128MB [ 0.145629] gcov: version magic: 0x41383552 [ 0.150151] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155285] pinctrl core: initialized pinctrl subsystem [ 0.158193] [ 0.158676] ************************************************************* [ 0.160012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162014] ** ** [ 0.165013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166011] ** ** [ 0.168014] ** This means that this kernel is built to expose internal ** [ 0.171013] ** IOMMU data structures, which may compromise security on ** [ 0.173012] ** your system. ** [ 0.175012] ** ** [ 0.178013] ** If you see this message and you are not debugging the ** [ 0.180013] ** kernel, report this immediately to your vendor! ** [ 0.183018] ** ** [ 0.185007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186007] ************************************************************* [ 0.187532] NET: Registered protocol family 16 [ 0.190546] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200098] cpuidle: using governor menu [ 0.202981] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206698] PCI: Using configuration type 1 for base access [ 0.209120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220129] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223045] cryptd: max_cpu_qlen set to 1000 [ 0.224261] ACPI: Added _OSI(Module Device) [ 0.226013] ACPI: Added _OSI(Processor Device) [ 0.228022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.234846] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240542] ACPI: Interpreter enabled [ 0.242069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243012] ACPI: Using IOAPIC for interrupt routing [ 0.245098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249373] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.262055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.265015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274494] acpiphp: Slot [2] registered [ 0.276158] acpiphp: Slot [5] registered [ 0.277132] acpiphp: Slot [6] registered [ 0.279131] acpiphp: Slot [7] registered [ 0.281124] acpiphp: Slot [8] registered [ 0.282104] acpiphp: Slot [9] registered [ 0.284123] acpiphp: Slot [10] registered [ 0.285128] acpiphp: Slot [3] registered [ 0.287119] acpiphp: Slot [4] registered [ 0.288095] acpiphp: Slot [11] registered [ 0.290100] acpiphp: Slot [12] registered [ 0.291111] acpiphp: Slot [13] registered [ 0.293100] acpiphp: Slot [14] registered [ 0.295120] acpiphp: Slot [15] registered [ 0.296113] acpiphp: Slot [16] registered [ 0.298101] acpiphp: Slot [17] registered [ 0.299082] acpiphp: Slot [18] registered [ 0.301099] acpiphp: Slot [19] registered [ 0.303116] acpiphp: Slot [20] registered [ 0.304088] acpiphp: Slot [21] registered [ 0.305087] acpiphp: Slot [22] registered [ 0.307073] acpiphp: Slot [23] registered [ 0.308024] acpiphp: Slot [24] registered [ 0.309092] acpiphp: Slot [25] registered [ 0.310094] acpiphp: Slot [26] registered [ 0.312170] acpiphp: Slot [27] registered [ 0.314109] acpiphp: Slot [28] registered [ 0.316105] acpiphp: Slot [29] registered [ 0.317100] acpiphp: Slot [30] registered [ 0.319152] acpiphp: Slot [31] registered [ 0.320060] PCI host bridge to bus 0000:00 [ 0.322019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.329025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.336031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338171] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341066] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345323] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.362050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.364019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.367020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372461] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.375911] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.378048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380667] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.402015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.407012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.411895] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.420015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.426013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.443015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.452552] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.466011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.508312] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.513012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.518013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.539014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.547958] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.553948] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.560013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.577015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.590111] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.601011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.611014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.632013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.644846] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.653014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.660014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.677029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.687628] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.690390] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.693378] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.695798] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.699240] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.705032] iommu: Default domain type: Passthrough [ 0.707391] SCSI subsystem initialized [ 0.709149] ACPI: bus type USB registered [ 0.710122] usbcore: registered new interface driver usbfs [ 0.712073] usbcore: registered new interface driver hub [ 0.714077] usbcore: registered new device driver usb [ 0.717180] pps_core: LinuxPPS API ver. 1 registered [ 0.719010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.722058] PTP clock support registered [ 0.724130] EDAC MC: Ver: 3.0.0 [ 0.726135] PCI: Using ACPI for IRQ routing [ 0.729029] NetLabel: Initializing [ 0.730010] NetLabel: domain hash size = 128 [ 0.732013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.735096] NetLabel: unlabeled traffic allowed by default [ 0.737134] vgaarb: loaded [ 0.739356] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.747328] clocksource: Switched to clocksource kvm-clock [ 0.852446] VFS: Disk quotas dquot_6.6.0 [ 0.853989] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.856601] *** VALIDATE ramfs *** [ 0.858376] *** VALIDATE hugetlbfs *** [ 0.860072] pnp: PnP ACPI init [ 0.862454] pnp: PnP ACPI: found 6 devices [ 0.881846] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.885136] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.887326] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.889683] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.892380] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.895422] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.899075] NET: Registered protocol family 2 [ 0.901350] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.906564] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.909899] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.915058] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.918087] TCP: Hash tables configured (established 65536 bind 65536) [ 0.922088] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.925210] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.928288] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.931812] NET: Registered protocol family 1 [ 0.936219] RPC: Registered named UNIX socket transport module. [ 0.938668] RPC: Registered udp transport module. [ 0.940725] RPC: Registered tcp transport module. [ 0.943471] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.945911] NET: Registered protocol family 44 [ 0.947725] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.949823] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.952079] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.955895] PCI: CLS 0 bytes, default 64 [ 0.957901] Unpacking initramfs... [ 2.361308] debug: unmapping init [mem 0xffff9f687cc54000-0xffff9f687ffbffff] [ 2.366163] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.368428] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.371653] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.873810] Initialise system trusted keyrings [ 2.875729] Key type blacklist registered [ 2.877876] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.887744] zbud: loaded [ 2.891350] *** VALIDATE nfs *** [ 2.892704] *** VALIDATE nfs4 *** [ 2.894450] pstore: using deflate compression [ 2.898402] Platform Keyring initialized [ 2.991587] NET: Registered protocol family 38 [ 2.994369] Key type asymmetric registered [ 2.995951] Asymmetric key parser 'x509' registered [ 2.998267] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.002064] io scheduler mq-deadline registered [ 3.003742] io scheduler kyber registered [ 3.005677] io scheduler bfq registered [ 3.009091] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.012675] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.015817] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.019479] ACPI: Power Button [PWRF] [ 3.025290] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.034659] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.051508] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.059526] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.078601] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.106750] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.136777] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.142540] Non-volatile memory driver v1.3 [ 3.143961] Linux agpgart interface v0.103 [ 3.174254] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.176922] vda: detected capacity change from 0 to 76775424 [ 3.193448] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.197230] vdb: detected capacity change from 0 to 1073741824 [ 3.213933] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.217277] vdc: detected capacity change from 0 to 2621440000 [ 3.232587] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.235740] vdd: detected capacity change from 0 to 2621440000 [ 3.251080] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.254411] vde: detected capacity change from 0 to 4294967296 [ 3.269169] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.272897] vdf: detected capacity change from 0 to 4294967296 [ 3.280653] libphy: Fixed MDIO Bus: probed [ 3.286540] usbcore: registered new interface driver usbserial_generic [ 3.289044] usbserial: USB Serial support registered for generic [ 3.291497] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.295642] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.297576] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.300374] mousedev: PS/2 mouse device common for all mice [ 3.303198] rtc_cmos 00:05: RTC can wake from S4 [ 3.306388] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.306862] rtc_cmos 00:05: registered as rtc0 [ 3.312731] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.316745] intel_pstate: CPU model not supported [ 3.319868] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.324812] hid: raw HID events driver (C) Jiri Kosina [ 3.327425] usbcore: registered new interface driver usbhid [ 3.329915] usbhid: USB HID core driver [ 3.330174] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.331675] drop_monitor: Initializing network drop monitor service [ 3.338043] Initializing XFRM netlink socket [ 3.340381] NET: Registered protocol family 10 [ 3.343520] Segment Routing with IPv6 [ 3.345339] NET: Registered protocol family 17 [ 3.347881] mpls_gso: MPLS GSO support [ 3.354466] RAS: Correctable Errors collector initialized. [ 3.356324] AVX version of gcm_enc/dec engaged. [ 3.357813] AES CTR mode by8 optimization enabled [ 3.435541] sched_clock: Marking stable (3435510277, 0)->(4313189861, -877679584) [ 3.439163] registered taskstats version 1 [ 3.441767] Loading compiled-in X.509 certificates [ 3.443470] zswap: loaded using pool lzo/zbud [ 3.466846] Key type big_key registered [ 3.481098] Key type encrypted registered [ 3.482843] ima: No TPM chip found, activating TPM-bypass! [ 3.485673] ima: Allocated hash algorithm: sha1 [ 3.487546] ima: No architecture policies found [ 3.489583] evm: Initialising EVM extended attributes: [ 3.491796] evm: security.selinux [ 3.493299] evm: security.ima [ 3.494260] evm: security.capability [ 3.495418] evm: HMAC attrs: 0x1 [ 3.497816] rtc_cmos 00:05: setting system clock to 2026-09-07 19:02:42 UTC (1788807762) [ 3.503309] debug: unmapping init [mem 0xffffffff93a03000-0xffffffff93bfffff] [ 3.505864] debug: unmapping init [mem 0xffffffff92782000-0xffffffff92a58fff] [ 3.515092] Write protecting the kernel read-only data: 28672k [ 3.518893] debug: unmapping init [mem 0xffffffff90e03000-0xffffffff90ffffff] [ 3.522019] debug: unmapping init [mem 0xffffffff91714000-0xffffffff917fffff] [ 3.560851] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.568596] systemd[1]: Detected virtualization kvm. [ 3.570161] systemd[1]: Detected architecture x86-64. [ 3.571888] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.601439] systemd[1]: No hostname configured. [ 3.603352] systemd[1]: Set hostname to . [ 3.605706] random: systemd: uninitialized urandom read (16 bytes read) [ 3.608609] systemd[1]: Initializing machine ID from random generator. [ 3.664358] random: ln: uninitialized urandom read (6 bytes read) [ 3.745165] random: systemd: uninitialized urandom read (16 bytes read) [ 3.748699] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.753363] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.758685] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.378418] device-mapper: uevent: version 1.0.3 [ 4.380924] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.079634] random: fast init done [ 5.110337] virtio_net virtio0 ens2: renamed from eth0 [ 5.129259] scsi host0: ata_piix [ 5.166328] scsi host1: ata_piix [ 5.168951] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.171860] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.005229] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.006157] random: crng init done [ 10.007876] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.485933] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.689768] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.952089] SELinux: Disabled at runtime. [ 12.013774] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.023855] systemd[1]: Detected virtualization kvm. [ 12.026114] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.536629] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.540649] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.545533] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.551395] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.555714] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.563836] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.570037] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ 12.597368] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.480044] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.105137] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.184747] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.479895] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.530367] EDAC sbridge: Ver: 1.1.2 [ 17.921114] Key type dns_resolver registered [ 18.463323] NFS: Registering the id_resolver key type [ 18.465480] Key type id_resolver registered [ 18.469738] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg308-server login: [ 62.309644] libcfs: loading out-of-tree module taints kernel. [ 62.357546] Key type ._llcrypt registered [ 62.359927] Key type .llcrypt registered [ 62.455437] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 76.585518] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 77.692965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 77.711382] alg: No test for adler32 (adler32-zlib) [ 79.037638] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 79.648687] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 81.329089] Key type lgssc registered [ 82.554891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.308717] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.629074] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 134.667148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 134.693899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 135.867804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 135.906588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 136.004893] Lustre: lustre-MDT0000: new disk, initializing [ 136.060161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 136.077409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 139.745725] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.273356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.362482] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 150.383525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 150.387244] Lustre: Skipped 1 previous similar message [ 150.444395] Lustre: lustre-MDT0001: new disk, initializing [ 150.507613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 150.529966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 150.545455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.424656] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.012443] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 163.928240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 164.171990] Lustre: lustre-OST0000: new disk, initializing [ 164.175372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 164.180567] Lustre: 8443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 164.231420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.202486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 167.212095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 167.286407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 168.528417] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.140730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 180.221873] Lustre: lustre-OST0001: new disk, initializing [ 180.225193] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 180.233726] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.267941] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.096725] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.897704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 185.907592] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 185.951046] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 195.238148] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 202.104447] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 206.312671] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing check_logdir /tmp/testlogs/ [ 209.867131] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing yml_node [ 213.056217] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 214.904905] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 216.822367] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 218.003390] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 15:06:15 EDT 2026 [ 230.188938] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 231.370487] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 247.268411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 247.270674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 247.285767] Lustre: Skipped 3 previous similar messages [ 247.291693] Lustre: Skipped 3 previous similar messages [ 251.406372] Lustre: server umount lustre-MDT0000 complete [ 253.956697] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808012 with bad export cookie 14320170481990007328 [ 253.960293] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 253.971840] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 257.508066] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.510102] LustreError: 6513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 257.513609] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 257.521239] Lustre: Skipped 1 previous similar message [ 257.533602] LustreError: 6513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 258.529141] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 258.529141] LustreError: 6515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.542270] Lustre: Skipped 2 previous similar messages [ 263.649615] LustreError: 9524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 263.650142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 263.664822] LustreError: 9524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 263.671800] Lustre: Skipped 1 previous similar message [ 268.257260] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 268.413878] Lustre: server umount lustre-MDT0001 complete [ 271.057182] Lustre: server umount lustre-OST0000 complete [ 273.740282] Lustre: server umount lustre-OST0001 complete [ 278.473595] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 285.263661] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 309.667612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 309.825440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 309.847238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 309.902423] Lustre: lustre-MDT0000: new disk, initializing [ 309.961836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 309.994303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 313.291352] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 321.038241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 321.142775] Lustre: 15059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 321.222711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 321.229219] Lustre: Skipped 1 previous similar message [ 321.299910] Lustre: lustre-MDT0001: new disk, initializing [ 321.397288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 321.437166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 321.447248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 324.750178] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 332.322781] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 336.503926] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 341.696213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 341.958751] Lustre: lustre-OST0000: new disk, initializing [ 341.966847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 341.974722] Lustre: 17029:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 342.054559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 343.156773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 343.167913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 343.257559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 346.833898] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 354.711891] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 358.267437] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 358.413552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 361.655730] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 361.823576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 364.006116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 364.012139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.019186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 364.020765] Lustre: Skipped 1 previous similar message [ 369.464291] Lustre: server umount lustre-OST0000 complete [ 372.193379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.202396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 372.205091] Lustre: Skipped 3 previous similar messages [ 378.462411] Lustre: server umount lustre-MDT0000 complete [ 381.392343] LustreError: 15054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808140 with bad export cookie 14320170481990008861 [ 381.400243] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.404053] LustreError: 15054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 381.677532] Lustre: server umount lustre-MDT0001 complete [ 385.964320] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:09:03 (1788808143) [ 399.820150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.132619] LustreError: 19903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 400.226970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.545160] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 405.478311] LustreError: 19904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 409.636724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.531190] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 419.332555] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 423.664083] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 425.363961] Lustre: Failing over lustre-MDT0000 [ 425.441859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 425.450066] LustreError: Skipped 1 previous similar message [ 425.454843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.465843] Lustre: Skipped 1 previous similar message [ 425.470120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 425.474725] Lustre: Skipped 2 previous similar messages [ 425.537296] Lustre: server umount lustre-MDT0000 complete [ 430.561373] LustreError: 19903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.575481] LustreError: 19903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 441.918966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.997063] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 444.965313] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 447.463412] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 447.465676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 447.493278] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 451.393587] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 455.529201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.547567] Lustre: Skipped 2 previous similar messages [ 460.759383] Lustre: server umount lustre-MDT0000 complete [ 460.775991] LustreError: 19900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 460.783881] LustreError: 19900:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 463.369302] LustreError: 19885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808222 with bad export cookie 14320170481990010275 [ 463.375229] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 463.378168] LustreError: 19885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 463.603937] Lustre: server umount lustre-MDT0001 complete [ 468.168339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.454962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.460093] Lustre: Skipped 2 previous similar messages [ 470.632705] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 475.401017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 477.776235] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 481.462351] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 484.165634] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 485.288132] Lustre: Failing over lustre-MDT0000 [ 485.295921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.301036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.302366] Lustre: Skipped 1 previous similar message [ 485.321306] Lustre: Skipped 3 previous similar messages [ 485.450583] Lustre: server umount lustre-MDT0000 complete [ 496.610556] LustreError: 23237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.619246] LustreError: 23237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 500.470609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.564435] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 501.737837] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 501.741666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 501.752613] Lustre: Skipped 1 previous similar message [ 501.764595] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 503.362476] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 509.277976] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 518.317915] Lustre: server umount lustre-MDT0000 complete [ 520.473101] LustreError: 23218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808279 with bad export cookie 14320170481990011213 [ 520.475513] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.481462] LustreError: 23218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 520.666046] Lustre: server umount lustre-MDT0001 complete [ 525.770917] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:11:23 (1788808283) [ 535.718196] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 542.404985] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 567.060150] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 573.566494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.739269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 573.767812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 573.826797] Lustre: lustre-MDT0000: new disk, initializing [ 573.892125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 576.445957] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 582.698264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.764594] Lustre: 29597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 582.962247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 585.282443] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 591.164605] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 594.206192] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 598.084121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 598.252846] Lustre: 31566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 598.301245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 598.307036] Lustre: Skipped 4 previous similar messages [ 600.101952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 600.124760] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 600.174579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 600.188861] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 601.796531] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 607.288622] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 612.056949] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 623.636631] Lustre: server umount lustre-OST0000 complete [ 625.636621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.641306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 625.642123] Lustre: Skipped 3 previous similar messages [ 625.649205] Lustre: Skipped 2 previous similar messages [ 635.935169] Lustre: server umount lustre-MDT0000 complete [ 637.886477] LustreError: 29590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808396 with bad export cookie 14320170481990011619 [ 637.889183] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.893646] LustreError: 29590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 647.232446] Lustre: server umount lustre-MDT0001 complete [ 652.186291] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 653.471298] Key type lgssc unregistered [ 653.617533] LNet: 33519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.621425] LNetError: 33519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.635304] LNet: Removed LNI 192.168.203.108@tcp [ 654.065136] Key type .llcrypt unregistered [ 654.067111] Key type ._llcrypt unregistered [ 666.468502] Key type ._llcrypt registered [ 666.470484] Key type .llcrypt registered [ 666.544429] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 667.001617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.028056] alg: No test for adler32 (adler32-zlib) [ 667.945789] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 668.076638] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 669.688461] Key type lgssc registered [ 670.287447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 674.188601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 674.199432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.475818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 677.742741] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 681.921904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 682.110638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 683.971861] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 687.350784] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 690.131246] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 693.454689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 693.642796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.502764] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 699.825731] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 708.012684] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 715.945567] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:14:34 (1788808474) [ 716.741650] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 717.559487] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 720.511936] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:14:38 (1788808478) [ 724.464629] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:14:42 (1788808482) [ 724.938822] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 728.832645] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 728.837380] Lustre: Skipped 2 previous similar messages [ 729.654644] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:14:47 (1788808487) [ 733.297326] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 733.300959] Lustre: Skipped 4 previous similar messages [ 740.279721] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:14:58 (1788808498) [ 741.538855] Lustre: *** cfs_fail_loc=131b, val=2*** [ 743.185513] Lustre: *** cfs_fail_loc=131b, val=2*** [ 746.960648] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:15:05 (1788808505) [ 747.790780] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 747.793979] Lustre: Skipped 9 previous similar messages [ 753.496512] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:15:11 (1788808511) [ 762.339041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 762.346768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.361038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 764.385693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 764.389592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 764.394156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.742292] Lustre: server umount lustre-OST0000 complete [ 768.994046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 768.999515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.008847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 769.013058] Lustre: Skipped 1 previous similar message [ 773.631888] Lustre: server umount lustre-MDT0000 complete [ 774.628980] LustreError: 34907:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 774.650218] LustreError: 34907:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 775.806520] LustreError: 34888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808534 with bad export cookie 18421332146347246169 [ 775.812864] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 775.816585] LustreError: 34888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 775.977912] Lustre: server umount lustre-MDT0001 complete [ 781.942672] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 783.666477] Key type lgssc unregistered [ 783.856668] LNet: 42762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.862775] LNetError: 42762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.874678] LNet: Removed LNI 192.168.203.108@tcp [ 784.353431] Key type .llcrypt unregistered [ 784.355642] Key type ._llcrypt unregistered [ 793.859640] Key type ._llcrypt registered [ 793.861371] Key type .llcrypt registered [ 793.908500] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 802.304062] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 802.823370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 802.876287] alg: No test for adler32 (adler32-zlib) [ 803.827625] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 803.959578] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 805.568402] Key type lgssc registered [ 806.141658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 824.710350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 824.736100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.870267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 825.893682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 825.937703] Lustre: lustre-MDT0000: new disk, initializing [ 825.976430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 825.998540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 828.195821] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 834.316212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.386935] Lustre: 45884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 834.425848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 834.429782] Lustre: Skipped 1 previous similar message [ 834.480439] Lustre: lustre-MDT0001: new disk, initializing [ 834.542683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 834.575159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 834.583930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 836.909908] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 842.314658] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 845.319877] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 848.632490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 848.786279] Lustre: lustre-OST0000: new disk, initializing [ 848.789415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 848.796691] Lustre: 47853:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 848.845169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 851.904433] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 857.121397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 857.128957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 857.206848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 857.703630] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 860.334492] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 860.476698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 863.019316] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 863.141804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 867.301214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.304067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 867.304807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 867.313080] Lustre: Skipped 1 previous similar message [ 867.325358] Lustre: Skipped 1 previous similar message [ 870.255691] Lustre: server umount lustre-OST0000 complete [ 872.420491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.423212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 872.429257] Lustre: Skipped 1 previous similar message [ 872.436240] Lustre: Skipped 1 previous similar message [ 877.537291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 877.541705] Lustre: Skipped 1 previous similar message [ 878.219392] Lustre: server umount lustre-MDT0000 complete [ 880.018609] LustreError: 45878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808639 with bad export cookie 16410664193890964895 [ 880.022267] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 880.025619] LustreError: 45878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 880.188542] Lustre: server umount lustre-MDT0001 complete [ 885.013410] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:17:23 (1788808643) [ 888.796805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 889.099370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.478882] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 894.433945] LustreError: 50307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 894.444111] LustreError: 50307:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 895.912330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.124206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 898.480528] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 902.384456] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 905.315754] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 909.035645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 909.251579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.691513] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 916.652642] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 923.736917] LustreError: 52835:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 924.326805] Lustre: Modifying parameter general.timeout=1 in log params [ 925.678178] Lustre: Modifying parameter general.timeout=4 in log params [ 925.682138] Lustre: Skipped 2 previous similar messages [ 928.122728] Lustre: Modifying parameter general.timeout=9 in log params [ 928.125707] Lustre: Skipped 4 previous similar messages [ 932.485506] Lustre: Modifying parameter general.timeout=18 in log params [ 932.488693] Lustre: Skipped 8 previous similar messages [ 940.651468] Lustre: Modifying parameter general.timeout=36 in log params [ 940.655522] Lustre: Skipped 17 previous similar messages [ 957.007449] Lustre: Modifying parameter general.timeout=73 in log params [ 957.010822] Lustre: Skipped 36 previous similar messages [ 987.111166] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:19:05 (1788808745) [ 988.506695] Lustre: 59749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 989.905621] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 989.911054] Lustre: Skipped 60 previous similar messages [ 990.367468] Lustre: 59945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 990.377670] Lustre: 59945:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 993.332623] Lustre: 60240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 993.340348] Lustre: 60240:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1007.185667] Lustre: 61612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1007.191314] LustreError: 61612:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1009.195638] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:19:27 (1788808767) [ 1014.753640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.755137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1014.760663] Lustre: Skipped 2 previous similar messages [ 1014.767128] Lustre: Skipped 2 previous similar messages [ 1019.874635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.880267] Lustre: Skipped 2 previous similar messages [ 1020.914282] Lustre: server umount lustre-MDT0000 complete [ 1022.684434] LustreError: 50288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808781 with bad export cookie 16410664193890965777 [ 1022.690262] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1022.691705] LustreError: 50288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1022.816738] Lustre: server umount lustre-MDT0001 complete [ 1034.795995] Lustre: server umount lustre-OST0000 complete [ 1046.801973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.929797] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1047.074105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.885295] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1052.968928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1053.025489] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1053.041122] Lustre: 64138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1055.086777] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1058.540479] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1061.171749] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1064.473374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1064.588459] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1067.697642] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1071.069746] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1077.257374] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1077.262467] Lustre: Skipped 19 previous similar messages [ 1077.281555] Lustre: 66704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1082.339719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1082.342127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.345646] LustreError: Skipped 1 previous similar message [ 1082.353333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1082.359139] Lustre: Skipped 1 previous similar message [ 1086.123148] Lustre: server umount lustre-OST0000 complete [ 1088.994835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1089.002745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.012474] Lustre: Skipped 1 previous similar message [ 1102.304126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1102.408356] Lustre: server umount lustre-MDT0000 complete [ 1102.817353] LustreError: 65145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.828823] LustreError: 65145:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1104.358529] LustreError: 66000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808863 with bad export cookie 16410664193890968318 [ 1104.362254] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1104.370993] LustreError: 66000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1104.541970] Lustre: server umount lustre-MDT0001 complete [ 1109.755275] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1111.153683] Key type lgssc unregistered [ 1111.315508] LNet: 67912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.320894] LNetError: 67912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.330790] LNet: Removed LNI 192.168.203.108@tcp [ 1111.740835] Key type .llcrypt unregistered [ 1111.742781] Key type ._llcrypt unregistered [ 1124.289808] Key type ._llcrypt registered [ 1124.291768] Key type .llcrypt registered [ 1124.344378] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1124.713238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1124.724786] alg: No test for adler32 (adler32-zlib) [ 1125.642280] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 1125.767599] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1127.360357] Key type lgssc registered [ 1127.924105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1131.447669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1131.461793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.690894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.494373] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1138.610208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.954358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1141.050064] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1144.590049] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1147.043324] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1149.976915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1150.252043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1152.880249] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1156.496947] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1170.142895] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1173.985364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.988737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1173.992051] Lustre: Skipped 2 previous similar messages [ 1173.997535] Lustre: Skipped 2 previous similar messages [ 1179.109203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.111653] Lustre: Skipped 2 previous similar messages [ 1179.777553] Lustre: server umount lustre-MDT0000 complete [ 1181.670440] LustreError: 69281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808940 with bad export cookie 1058724164236895668 [ 1181.674585] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1181.676058] LustreError: 69281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1181.681373] LustreError: 69292:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1181.834042] Lustre: server umount lustre-MDT0001 complete [ 1193.915321] Lustre: server umount lustre-OST0000 complete [ 1205.782510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.953033] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1205.970401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1206.095887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.924869] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1211.874718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.928187] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1211.950043] Lustre: 74182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1212.202041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1213.929368] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1217.282222] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1219.903877] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1223.326392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1223.440722] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1223.643656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.371320] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1229.712538] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1236.118641] Lustre: 76757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1237.098329] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1242.084391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1242.084537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.085780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1242.089423] LustreError: Skipped 1 previous similar message [ 1242.094956] Lustre: Skipped 1 previous similar message [ 1245.539514] Lustre: server umount lustre-OST0000 complete [ 1247.201256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.204163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.208570] Lustre: Skipped 1 previous similar message [ 1247.213530] Lustre: Skipped 1 previous similar message [ 1252.323977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.327234] Lustre: Skipped 2 previous similar messages [ 1253.251608] Lustre: server umount lustre-MDT0000 complete [ 1255.049231] LustreError: 74173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809014 with bad export cookie 1058724164236896466 [ 1255.052377] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.055867] LustreError: 74173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1255.212186] Lustre: server umount lustre-MDT0001 complete [ 1260.195864] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1261.456189] Key type lgssc unregistered [ 1261.616504] LNet: 77962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.620866] LNetError: 77962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1261.631593] LNet: Removed LNI 192.168.203.108@tcp [ 1262.038464] Key type .llcrypt unregistered [ 1262.040972] Key type ._llcrypt unregistered [ 1273.797635] Key type ._llcrypt registered [ 1273.799341] Key type .llcrypt registered [ 1273.864854] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1274.390859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1274.406636] alg: No test for adler32 (adler32-zlib) [ 1275.300722] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 1275.436269] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1277.041472] Key type lgssc registered [ 1277.598936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1281.574191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1281.589847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.859823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.193544] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1289.416917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.836974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1291.815507] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1295.405550] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1298.132990] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1302.083380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1302.404496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1305.425110] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1309.312733] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1317.696942] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1318.499927] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1318.502793] Lustre: Skipped 1 previous similar message [ 1319.670463] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1319.673496] Lustre: Skipped 2 previous similar messages [ 1321.992741] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1321.996682] Lustre: Skipped 5 previous similar messages [ 1326.266716] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1326.271993] Lustre: Skipped 10 previous similar messages [ 1334.431748] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1334.435476] Lustre: Skipped 20 previous similar messages [ 1350.690054] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1350.694370] Lustre: Skipped 40 previous similar messages [ 1359.943081] Lustre: 87416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1361.175327] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:25:19 (1788809119) [ 1361.814497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1362.523526] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:25:20 (1788809120) [ 1363.937237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1363.940876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1363.948168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1365.986477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1365.988781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1365.989353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1366.005159] Lustre: Skipped 1 previous similar message [ 1371.108420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1371.111660] Lustre: Skipped 1 previous similar message [ 1376.227721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1376.231040] Lustre: Skipped 1 previous similar message [ 1378.272126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1378.343411] Lustre: server umount lustre-OST0000 complete [ 1381.346324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.347063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1381.351823] Lustre: Skipped 1 previous similar message [ 1391.585540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1391.589253] Lustre: Skipped 4 previous similar messages [ 1394.144592] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1394.218262] Lustre: server umount lustre-MDT0000 complete [ 1395.742466] LustreError: 80091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809154 with bad export cookie 7272394120350119819 [ 1395.745887] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1395.747861] LustreError: 80091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1395.892562] Lustre: server umount lustre-MDT0001 complete [ 1400.180741] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1401.299852] Key type lgssc unregistered [ 1401.443329] LNet: 88664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.446664] LNetError: 88664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.453500] LNet: Removed LNI 192.168.203.108@tcp [ 1401.788569] Key type .llcrypt unregistered [ 1401.790232] Key type ._llcrypt unregistered [ 1403.330061] Key type ._llcrypt registered [ 1403.331575] Key type .llcrypt registered [ 1403.379703] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1405.214093] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules [ 1405.622760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1405.675069] alg: No test for adler32 (adler32-zlib) [ 1406.558158] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 1406.671408] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1408.288218] Key type lgssc registered [ 1408.764887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1409.223887] LustreError: 89812:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 1417.652684] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1421.506474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1430.383522] LustreError: 89812:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1430.397854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.666631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.402354] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1436.960202] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:26:35 (1788809195) [ 1438.837190] LustreError: 90870:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1438.843612] LustreError: 90870:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1438.848230] LustreError: 90870:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1438.991719] Lustre: server umount lustre-MDT0000 complete [ 1444.740406] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1446.181558] Key type lgssc unregistered [ 1446.358786] LNet: 92361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.366221] LNetError: 92361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1446.378264] LNet: Removed LNI 192.168.203.108@tcp [ 1446.836304] Key type .llcrypt unregistered [ 1446.839649] Key type ._llcrypt unregistered [ 1455.711519] Key type ._llcrypt registered [ 1455.713153] Key type .llcrypt registered [ 1455.765846] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1456.135704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1456.153090] alg: No test for adler32 (adler32-zlib) [ 1457.027578] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 1457.134502] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1458.736196] Key type lgssc registered [ 1459.261623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1463.065995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1463.077748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.324591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1466.165572] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1469.983154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.344496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1472.194656] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1475.438235] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1477.903638] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1480.924243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1481.215220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.820049] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1487.476569] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1522.268895] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:28:00 (1788809280) [ 1523.002145] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1523.743946] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:28:02 (1788809282) [ 1526.245912] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.247357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1526.254060] Lustre: Skipped 2 previous similar messages [ 1526.258752] Lustre: Skipped 1 previous similar message [ 1531.224935] Lustre: server umount lustre-MDT0000 complete [ 1531.362447] LustreError: 93553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.370602] LustreError: 93553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1532.701276] LustreError: 95404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809291 with bad export cookie 8147485615032850041 [ 1532.704156] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1532.706435] LustreError: 95404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1532.846773] Lustre: server umount lustre-MDT0001 complete [ 1544.747157] Lustre: server umount lustre-OST0000 complete [ 1550.018689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1550.217807] LustreError: 97840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.305118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.048645] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1555.427051] LustreError: 97841:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.339373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1556.682172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1558.477430] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1561.746941] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1564.317983] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1569.921695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1570.021285] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1570.024983] LustreError: 97830:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1570.037926] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1570.043308] LustreError: 99922:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 1570.046800] LustreError: 99922:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 1570.050151] LustreError: 99922:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1570.083353] Lustre: server umount lustre-OST0000 complete [ 1570.085232] LustreError: 99922:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1573.631580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1573.726760] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1573.732275] LustreError: 97830:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1573.738718] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1573.742849] LustreError: 100385:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 1573.747039] LustreError: 100385:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 1573.749925] LustreError: 100385:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1573.789243] Lustre: server umount lustre-OST0000 complete [ 1573.791132] LustreError: 100385:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1577.997540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1578.086831] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1578.090052] Lustre: Found index 0 for lustre-OST0000, updating log [ 1578.093508] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1578.125305] Lustre: lustre-OST0000: new disk, initializing [ 1578.129109] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1578.133330] Lustre: 100913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1578.274842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.718792] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1582.263428] Lustre: Failing over lustre-OST0000 [ 1582.321660] Lustre: server umount lustre-OST0000 complete [ 1584.039243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1584.043320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1584.047764] Lustre: Skipped 1 previous similar message [ 1589.218051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1589.220898] Lustre: Skipped 2 previous similar messages [ 1590.182922] Lustre: server umount lustre-MDT0000 complete [ 1591.700254] LustreError: 99943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809350 with bad export cookie 8147485615032852771 [ 1591.701547] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1591.706393] LustreError: 99943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1602.072906] Lustre: server umount lustre-MDT0001 complete [ 1605.852216] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:29:24 (1788809364) [ 1612.395630] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1616.721928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.908349] LustreError: 103259:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1616.957342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.691507] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1621.985242] LustreError: 103260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1622.414806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.609690] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1625.876540] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1628.613612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1628.937995] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1631.544323] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1632.040295] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1632.063408] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1632.065387] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1635.419422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1635.472088] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1635.519357] Lustre: lustre-OST0001: new disk, initializing [ 1635.522944] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1635.527733] Lustre: 105575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1638.582145] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1639.956953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1639.962580] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1640.014875] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1647.852595] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1654.243626] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1657.315622] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.317125] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1657.324675] Lustre: Skipped 2 previous similar messages [ 1657.327888] Lustre: Skipped 1 previous similar message [ 1662.190982] Lustre: server umount lustre-MDT0001 complete [ 1662.434322] LustreError: 103990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.441899] LustreError: 103990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1666.118828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1666.172449] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1666.175402] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1666.199122] Lustre: 103248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1666.535076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1666.538438] Lustre: Skipped 3 previous similar messages [ 1667.754961] LustreError: 107225:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1667.758894] LustreError: 107225:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1667.764306] LustreError: 103251:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1667.815312] LustreError: 107240:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.108@tcp: cfg command failed: rc = -17 [ 1667.820600] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1667.820600] [ 1668.410415] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1671.656375] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1672.660806] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 15:30:31 (1788809431) [ 1679.666562] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1687.671892] Lustre: 109181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1698.148636] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1699.696359] Lustre: 110312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1753.569543] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.570513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1753.570633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.570637] Lustre: Skipped 2 previous similar messages [ 1753.580638] Lustre: Skipped 1 previous similar message [ 1759.493178] Lustre: server umount lustre-MDT0000 complete [ 1761.047842] LustreError: 103239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809520 with bad export cookie 8147485615032853618 [ 1761.050074] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1761.053954] LustreError: 103239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1761.172185] Lustre: server umount lustre-MDT0001 complete [ 1772.999192] Lustre: server umount lustre-OST0000 complete [ 1784.808945] Lustre: server umount lustre-OST0001 complete [ 1787.075101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1790.939732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1794.403760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1798.401695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1806.326853] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1810.663890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.677504] Lustre: lustre-MDT0000: reset Object Index mappings [ 1810.861520] LustreError: 114258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.870219] LustreError: 114258:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1810.904090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1812.602271] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1816.094050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.105959] Lustre: lustre-MDT0001: reset Object Index mappings [ 1818.179287] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1819.405390] Lustre: 115404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1822.052085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1824.818072] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1826.474508] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 1828.534644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1831.308611] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1833.968353] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1836.013291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 1836.033715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1840.568902] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1842.075071] Lustre: 117282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1863.966875] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 15:33:42 (1788809622) [ 1893.346696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1893.354746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.368310] Lustre: Skipped 1 previous similar message [ 1893.372267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1893.375918] Lustre: Skipped 7 previous similar messages [ 1898.690854] Lustre: server umount lustre-MDT0000 complete [ 1900.539508] LustreError: 117284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809659 with bad export cookie 8147485615032900441 [ 1900.542460] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.546586] LustreError: 117284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1901.538352] LustreError: 114254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.538543] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.546079] LustreError: 114254:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1901.550849] Lustre: Skipped 4 previous similar messages [ 1906.878496] Lustre: server umount lustre-MDT0001 complete [ 1914.836798] Lustre: server umount lustre-OST0000 complete [ 1922.737662] Lustre: server umount lustre-OST0001 complete [ 1925.494135] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 1929.685906] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1944.955755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.067719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1945.084398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1945.131582] Lustre: lustre-MDT0000: new disk, initializing [ 1945.166128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.171489] Lustre: Skipped 3 previous similar messages [ 1945.184463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1946.972323] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1950.782621] Lustre: server umount lustre-MDT0000 complete [ 1955.092619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.228363] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1955.238530] Lustre: 121540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1955.247467] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1957.083245] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1959.303089] Lustre: server umount lustre-MDT0000 complete [ 1962.905049] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:35:21 (1788809721) [ 1963.621204] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1964.405535] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:35:22 (1788809722) [ 1971.089509] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1974.711717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.474389] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1979.753659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.803687] Lustre: 123590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1979.832930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1979.835753] Lustre: Skipped 1 previous similar message [ 1979.876718] Lustre: lustre-MDT0001: new disk, initializing [ 1979.937379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1979.942741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1981.487853] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1984.010895] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1986.422394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1986.539568] Lustre: lustre-OST0000: new disk, initializing [ 1986.541837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1986.545372] Lustre: 125166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1987.030979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1987.035247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1987.077870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1988.905483] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1993.541796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1993.615363] Lustre: 126036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1994.961693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1994.967807] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1995.015382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1996.087240] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2000.971152] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2002.364568] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2139.558134] Lustre: 132411:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2139.561156] Lustre: 132411:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2139.563574] Lustre: 132411:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2139.567415] Lustre: 132411:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2139.570595] Lustre: 132411:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2139.573599] Lustre: 132411:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2232.366433] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:39:50 (1788809990) [ 2246.114393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2246.118740] LustreError: 125159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.120245] Lustre: Skipped 2 previous similar messages [ 2246.127737] LustreError: 125159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2246.133755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2246.166942] Lustre: server umount lustre-MDT0000 complete [ 2247.550246] LustreError: 123581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810006 with bad export cookie 8147485615032958205 [ 2247.554163] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.555456] LustreError: 123581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2251.236721] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2251.239125] Lustre: Skipped 9 previous similar messages [ 2253.788489] Lustre: server umount lustre-MDT0001 complete [ 2261.377771] Lustre: server umount lustre-OST0000 complete [ 2276.832210] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2276.905202] Lustre: server umount lustre-OST0001 complete [ 2279.306553] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 2282.973657] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2297.448533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.545589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2297.560463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2297.563249] Lustre: Skipped 1 previous similar message [ 2297.602292] Lustre: lustre-MDT0000: new disk, initializing [ 2297.605019] Lustre: Skipped 1 previous similar message [ 2297.645950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.649492] Lustre: Skipped 5 previous similar messages [ 2297.657304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2299.298516] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2304.085035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.138089] Lustre: 135427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2304.267430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2305.865733] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2310.164085] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2312.225759] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2314.829493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2314.947430] Lustre: lustre-OST0000: new disk, initializing [ 2314.949191] Lustre: Skipped 1 previous similar message [ 2314.951514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2314.953821] Lustre: Skipped 2 previous similar messages [ 2314.956710] Lustre: 137394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2316.289610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2316.293253] Lustre: Skipped 1 previous similar message [ 2316.295571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2316.341048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2317.238426] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2321.147308] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2324.352659] Lustre: lustre-MDD0000: changelog on [ 2325.261537] Lustre: lustre-MDD0001: changelog on [ 2768.792929] Lustre: lustre-MDD0001: changelog off [ 2769.426358] Lustre: 139781:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2769.750322] Lustre: lustre-MDD0000: changelog off [ 2771.939536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2771.939656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.942469] LustreError: Skipped 1 previous similar message [ 2771.946633] Lustre: Skipped 3 previous similar messages [ 2771.949159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.952904] Lustre: Skipped 3 previous similar messages [ 2777.433721] Lustre: server umount lustre-OST0000 complete [ 2780.640979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2784.894468] Lustre: server umount lustre-MDT0000 complete [ 2786.191366] LustreError: 135420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810545 with bad export cookie 8147485615033761175 [ 2786.195810] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2786.196442] LustreError: 135420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2786.330351] Lustre: server umount lustre-MDT0001 complete [ 2789.933112] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2790.961689] Key type lgssc unregistered [ 2791.081452] LNet: 145928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2791.084787] LNetError: 145928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2791.093492] LNet: Removed LNI 192.168.203.108@tcp [ 2791.411108] Key type .llcrypt unregistered [ 2791.412199] Key type ._llcrypt unregistered [ 2797.839960] Key type ._llcrypt registered [ 2797.841105] Key type .llcrypt registered [ 2797.874778] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 2804.169185] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2804.546201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2804.553955] alg: No test for adler32 (adler32-zlib) [ 2805.422134] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 2805.529064] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2807.112215] Key type lgssc registered [ 2807.509872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2819.805982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2819.813078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.914509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2820.927115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2820.965381] Lustre: lustre-MDT0000: new disk, initializing [ 2820.989568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.996408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2822.463684] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2826.689702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.732073] Lustre: 149098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2826.756495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2826.758300] Lustre: Skipped 1 previous similar message [ 2826.797617] Lustre: lustre-MDT0001: new disk, initializing [ 2826.831139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2826.846156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2826.850880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2828.312805] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2832.263122] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2834.218056] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2836.706171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2836.831438] Lustre: lustre-OST0000: new disk, initializing [ 2836.833558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2836.836757] Lustre: 151066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2836.876821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2839.046192] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2839.103571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2839.109290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2839.153578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2842.976189] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2845.287103] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_module obdecho/obdecho [ 2845.923130] LustreError: 152069:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2845.926738] LustreError: 152069:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2846.535845] LustreError: 152165:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 2852.320946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2852.323362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.328047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.883332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.883594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.888405] Lustre: Skipped 1 previous similar message [ 2854.890702] Lustre: Skipped 1 previous similar message [ 2855.975858] Lustre: server umount lustre-MDT0000 complete [ 2857.247277] LustreError: 149135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810616 with bad export cookie 11766437655088915156 [ 2857.248984] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.250795] LustreError: 149135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2857.359745] Lustre: server umount lustre-MDT0001 complete [ 2868.885480] Lustre: server umount lustre-OST0000 complete [ 2875.464128] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 2878.646618] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2890.225585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.315550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2890.325995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2890.364476] Lustre: lustre-MDT0000: new disk, initializing [ 2890.386080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.392607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2891.707144] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2895.668724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.710402] Lustre: 155874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2895.731491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2895.733215] Lustre: Skipped 1 previous similar message [ 2895.774063] Lustre: lustre-MDT0001: new disk, initializing [ 2895.805300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2895.818721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2895.823336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2897.088504] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2900.622757] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2902.382480] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2904.511056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2904.623431] Lustre: lustre-OST0000: new disk, initializing [ 2904.625570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2904.628907] Lustre: 157841:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2904.660381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.446497] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2909.689551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2909.692864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2909.727117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2910.018782] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2911.534653] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2911.615660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2913.076316] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2913.161285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2914.785649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2914.788268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2914.789247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2914.792046] Lustre: Skipped 1 previous similar message [ 2914.794404] Lustre: Skipped 1 previous similar message [ 2915.918909] Lustre: server umount lustre-OST0000 complete [ 2919.905733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.906613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2919.909759] Lustre: Skipped 1 previous similar message [ 2919.913238] Lustre: Skipped 1 previous similar message [ 2923.296459] Lustre: server umount lustre-MDT0000 complete [ 2924.561333] LustreError: 157840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810683 with bad export cookie 11766437655088916108 [ 2924.564922] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2924.565895] LustreError: 157840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2924.675079] Lustre: server umount lustre-MDT0001 complete [ 2925.903832] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:51:24 (1788810684) [ 2926.417757] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2926.962587] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:51:25 (1788810685) [ 2929.409105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.595383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.926609] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2933.803144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2935.314995] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2937.759927] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2939.593822] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2941.773821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2943.751545] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2945.939415] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2959.332464] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.334687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2959.336515] Lustre: Skipped 2 previous similar messages [ 2959.341291] Lustre: Skipped 2 previous similar messages [ 2964.257638] Lustre: server umount lustre-MDT0000 complete [ 2964.450327] LustreError: 162544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2964.456369] LustreError: 162544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2965.454954] LustreError: 160232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810724 with bad export cookie 11766437655088916976 [ 2965.458205] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2965.458658] LustreError: 160232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2966.536623] LustreError: 160274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.569378] LustreError: 160246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.569583] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.582050] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2969.584295] Lustre: Skipped 2 previous similar messages [ 2971.637534] Lustre: server umount lustre-MDT0001 complete [ 2973.814183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2978.218598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2978.314212] LustreError: 164109:0:(llog.c:1655:llog_backup()) MGC192.168.203.108@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2978.317653] Lustre: 164109:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.108@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2987.016831] LustreError: 164113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.022841] LustreError: 164113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2991.112586] LustreError: 164113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2991.117446] LustreError: 164113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3000.289062] LustreError: 164113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.295882] LustreError: 164113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3017.736385] LustreError: 164114:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.736385] LustreError: 164113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.736399] LustreError: 164113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 3039.328303] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3039.331388] Lustre: 164109:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.108@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3080.167327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3080.170499] Lustre: Skipped 2 previous similar messages [ 3081.656491] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3082.248336] LustreError: 164114:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.253865] LustreError: 164114:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 3084.689224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.285096] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3086.848742] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3089.285428] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3089.803360] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3090.407036] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3091.630826] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3093.176502] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 3099.745995] Lustre: server umount lustre-MDT0000 complete [ 3100.642509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.648347] Lustre: Skipped 2 previous similar messages [ 3100.649969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3100.654688] LustreError: Skipped 1 previous similar message [ 3101.008195] LustreError: 160230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810860 with bad export cookie 11766437655088923416 [ 3101.011778] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3101.012091] LustreError: 160230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3101.128614] Lustre: server umount lustre-MDT0001 complete [ 3112.666694] Lustre: server umount lustre-OST0000 complete [ 3119.964307] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 3123.327479] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3135.798592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3135.881074] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3135.892062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3135.927406] Lustre: lustre-MDT0000: new disk, initializing [ 3135.948938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3137.281105] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3141.284974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.322266] Lustre: 169180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3141.417206] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3142.755489] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3146.233187] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3147.926462] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3150.007827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3150.101096] Lustre: 171147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3150.126810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.129090] Lustre: Skipped 3 previous similar messages [ 3151.417373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3151.447779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3151.907986] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3155.468634] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3156.742019] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3156.816364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3158.172560] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3158.253731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3161.572244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3161.573545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.574196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3161.574204] Lustre: Skipped 1 previous similar message [ 3161.574796] LustreError: Skipped 1 previous similar message [ 3161.580291] Lustre: Skipped 1 previous similar message [ 3173.344137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3173.400393] Lustre: server umount lustre-OST0000 complete [ 3180.698234] Lustre: server umount lustre-MDT0000 complete [ 3181.983060] LustreError: 169972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810940 with bad export cookie 11766437655088940216 [ 3181.985311] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3181.986830] LustreError: 169972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3182.099813] Lustre: server umount lustre-MDT0001 complete [ 3184.871843] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:55:43 (1788810943) [ 3186.953690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3188.342421] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3191.049543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3192.502923] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3194.870504] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3196.575179] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3198.812790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3201.052119] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3203.462184] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3217.905060] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:56:16 (1788810976) [ 3222.502949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3222.504420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3222.505490] LustreError: Skipped 1 previous similar message [ 3222.511776] Lustre: Skipped 8 previous similar messages [ 3225.050342] Lustre: server umount lustre-OST0000 complete [ 3227.105106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.109311] Lustre: Skipped 4 previous similar messages [ 3232.595162] Lustre: server umount lustre-MDT0000 complete [ 3232.737234] LustreError: 173576:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.742194] LustreError: 173576:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 3233.919199] LustreError: 173584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788810992 with bad export cookie 11766437655088941084 [ 3233.922635] LustreError: 173584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3233.923020] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3234.048695] Lustre: server umount lustre-MDT0001 complete [ 3238.008402] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3239.196774] Key type lgssc unregistered [ 3239.348345] LNet: 177301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3239.351212] LNetError: 177301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3239.361340] LNet: Removed LNI 192.168.203.108@tcp [ 3239.706755] Key type .llcrypt unregistered [ 3239.708686] Key type ._llcrypt unregistered [ 3247.606889] Key type ._llcrypt registered [ 3247.607806] Key type .llcrypt registered [ 3247.655148] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3247.971445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3248.028420] alg: No test for adler32 (adler32-zlib) [ 3248.874885] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 3248.964933] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3250.544196] Key type lgssc registered [ 3250.934332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3253.644257] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3253.650890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3269.088361] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3274.208315] LustreError: 178517:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 3279.328596] LustreError: 178517:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3279.333654] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3279.336523] LustreError: MGC192.168.203.108@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3279.341930] LustreError: 178517:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3279.344699] LustreError: 178517:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 3279.346947] LustreError: 178517:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 3279.374376] Lustre: server umount lustre-OST0000 complete [ 3279.376116] LustreError: 178517:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3282.641371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.807100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.131868] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3288.064746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.205406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3289.623235] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3291.940389] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3293.766540] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3296.083823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3296.155491] LustreError: 180828:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3296.159469] Lustre: 180828:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.108@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3296.207159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3298.170811] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3300.606511] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3303.395342] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3303.397724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3303.401650] Lustre: Skipped 2 previous similar messages [ 3303.403842] Lustre: Skipped 1 previous similar message [ 3307.539867] Lustre: server umount lustre-MDT0000 complete [ 3308.514192] LustreError: 179014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3308.523168] LustreError: 179014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3309.848708] LustreError: 178996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811068 with bad export cookie 9674371155871489031 [ 3309.851528] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3309.855763] LustreError: 178996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3310.059264] Lustre: server umount lustre-MDT0001 complete [ 3322.528579] Lustre: server umount lustre-OST0000 complete [ 3327.049755] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 3333.707260] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3350.569120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.662503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3350.682655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3350.734124] Lustre: lustre-MDT0000: new disk, initializing [ 3350.782747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3350.797128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3352.824287] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3358.211379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.269578] Lustre: 184350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3358.302055] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3358.305437] Lustre: Skipped 1 previous similar message [ 3358.358125] Lustre: lustre-MDT0001: new disk, initializing [ 3358.417690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3358.434763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3358.439642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3360.598498] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3365.326500] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3367.648333] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3370.230694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3370.337806] Lustre: lustre-OST0000: new disk, initializing [ 3370.340564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3370.344648] Lustre: 186315:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3370.382203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.629852] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3376.918662] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3378.505259] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3378.686931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3378.691606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3378.724402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3379.614931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3381.168730] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3381.252602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3383.779240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3383.779329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.783158] LustreError: Skipped 1 previous similar message [ 3383.790056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3383.793266] Lustre: Skipped 2 previous similar messages [ 3388.129651] Lustre: server umount lustre-OST0000 complete [ 3394.017705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.018202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3394.022648] Lustre: Skipped 2 previous similar messages [ 3395.747975] Lustre: server umount lustre-MDT0000 complete [ 3397.060448] LustreError: 187660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811156 with bad export cookie 9674371155871489717 [ 3397.065101] LustreError: 187660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3397.065183] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3397.186408] Lustre: server umount lustre-MDT0001 complete [ 3400.024458] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:59:18 (1788811158) [ 3405.991016] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3407.049356] Key type lgssc unregistered [ 3407.188409] LNet: 189262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3407.192433] LNetError: 189262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3407.202421] LNet: Removed LNI 192.168.203.108@tcp [ 3407.528894] Key type .llcrypt unregistered [ 3407.530100] Key type ._llcrypt unregistered [ 3414.794724] Key type ._llcrypt registered [ 3414.796451] Key type .llcrypt registered [ 3414.834615] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3415.140383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3415.148289] alg: No test for adler32 (adler32-zlib) [ 3415.999515] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 3416.090345] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3417.672223] Key type lgssc registered [ 3418.055604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3420.895237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3420.902752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.092115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.616989] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3426.482417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.601034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3427.937948] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3430.548368] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3432.602834] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3435.469766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3435.555728] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3435.557527] LustreError: 192315:0:(llog.c:1696:llog_backup()) MGC192.168.203.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3435.561222] Lustre: 192315:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.108@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3435.609825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.702564] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3440.390031] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3442.146878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.147365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3442.147616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.151633] Lustre: Skipped 1 previous similar message [ 3447.265293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3447.268618] Lustre: Skipped 4 previous similar messages [ 3447.386485] Lustre: server umount lustre-MDT0000 complete [ 3448.797161] LustreError: 190432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811207 with bad export cookie 9558346521812839954 [ 3448.799384] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3448.801974] LustreError: 190432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3448.903353] Lustre: server umount lustre-MDT0001 complete [ 3460.655485] Lustre: server umount lustre-OST0000 complete [ 3464.176953] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 3467.455355] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3480.636342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.761982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3480.775755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3480.818647] Lustre: lustre-MDT0000: new disk, initializing [ 3480.856411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3480.865211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3482.516733] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3487.430084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.491375] Lustre: 195830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3487.517704] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3487.520375] Lustre: Skipped 1 previous similar message [ 3487.527770] LustreError: 196613:0:(llog.c:1696:llog_backup()) MGC192.168.203.108@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3487.533047] LustreError: 196613:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 3487.536299] Lustre: 196613:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.108@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3487.549822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3487.552909] Lustre: Skipped 1 previous similar message [ 3487.595078] Lustre: lustre-MDT0001: new disk, initializing [ 3487.653895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3487.675935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3487.681390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3489.273196] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3493.674799] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3495.968629] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3498.455943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3498.551386] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3498.552870] Lustre: Skipped 1 previous similar message [ 3498.554893] LustreError: 197776:0:(llog.c:1696:llog_backup()) MGC192.168.203.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3498.560079] LustreError: 197776:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 3498.563192] Lustre: 197776:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.108@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3498.567852] Lustre: 197776:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3498.578314] Lustre: lustre-OST0000: new disk, initializing [ 3498.581024] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3498.585779] Lustre: 197800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3498.629962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.841726] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3504.739885] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3506.294231] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3507.714751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3507.718815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3507.762982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3508.418610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3509.955287] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3510.031847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3512.802188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3512.802966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.805488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3512.805600] LustreError: Skipped 1 previous similar message [ 3512.810417] Lustre: Skipped 2 previous similar messages [ 3516.816174] Lustre: server umount lustre-OST0000 complete [ 3518.432842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3518.435162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.440436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3518.442392] Lustre: Skipped 1 previous similar message [ 3523.041035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3523.041559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.250100] Lustre: server umount lustre-MDT0000 complete [ 3525.505250] LustreError: 195824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811284 with bad export cookie 9558346521812840640 [ 3525.508196] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3525.510334] LustreError: 195824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3525.625910] Lustre: server umount lustre-MDT0001 complete [ 3528.621875] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:01:27 (1788811287) [ 3534.239978] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3537.688349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.828983] LustreError: 200947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.837161] LustreError: 200947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3537.845416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3539.260504] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3542.200030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.707295] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3544.696869] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3546.994051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3549.139730] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3551.979095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3552.043443] Lustre: lustre-OST0001: new disk, initializing [ 3552.045162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3552.048229] Lustre: 203255:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3553.541267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3553.547399] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3553.588684] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3554.148162] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3559.576501] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3564.287243] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3564.955618] Lustre: DEBUG MARKER: ADD OST3 [ 3566.774793] loop: module loaded [ 3566.799974] loop0: detected capacity change from 0 to 204800000 [ 3566.940064] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3566.944583] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3566.951619] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3569.071183] loop0: detected capacity change from 0 to 204800000 [ 3570.997596] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3571.080150] Lustre: lustre-OST0003: new disk, initializing [ 3571.083829] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3571.088783] Lustre: 204984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3571.144499] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3571.147428] Lustre: Skipped 3 previous similar messages [ 3572.022275] Lustre: DEBUG MARKER: STOP OST3 [ 3572.187055] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3572.233247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3572.237180] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3572.250332] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3572.730116] Lustre: Failing over lustre-OST0003 [ 3572.782986] Lustre: server umount lustre-OST0003 complete [ 3573.411605] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3574.240357] LustreError: 205204:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3577.313183] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3577.317276] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3577.320981] LustreError: 203500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3577.322912] Lustre: Skipped 1 previous similar message [ 3577.328649] Lustre: DEBUG MARKER: START OST3 again [ 3579.741113] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3579.832722] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3581.138084] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3581.867405] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3581.867458] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3581.913113] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3595.886892] Lustre: Failing over lustre-OST0003 [ 3595.930839] Lustre: server umount lustre-OST0003 complete [ 3597.281757] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3597.281895] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.285553] LustreError: Skipped 1 previous similar message [ 3597.290680] LustreError: 206980:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.298615] LustreError: 206980:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3600.357141] LustreError: 202432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3600.363018] LustreError: 202432:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3605.475979] LustreError: 202430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3605.483790] LustreError: 202430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3609.056848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3609.060944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3609.063819] Lustre: Skipped 1 previous similar message [ 3614.621679] Lustre: server umount lustre-MDT0000 complete [ 3615.713554] LustreError: 204091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3615.720267] LustreError: 204091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3615.967273] LustreError: 200929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811374 with bad export cookie 9558346521812841508 [ 3615.975151] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3620.833934] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.838962] Lustre: Skipped 6 previous similar messages [ 3622.192803] Lustre: server umount lustre-MDT0001 complete [ 3623.592521] Lustre: server umount lustre-OST0000 complete [ 3624.926547] Lustre: server umount lustre-OST0001 complete [ 3627.101660] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 3630.033592] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3643.857810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3643.953611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3643.966399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3644.003716] Lustre: lustre-MDT0000: new disk, initializing [ 3644.029812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3644.033886] Lustre: Skipped 1 previous similar message [ 3644.041103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3645.482978] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3649.869983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.917309] Lustre: 210392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3650.027973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3651.506239] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3655.401031] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3657.355702] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3659.608643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3659.720107] Lustre: 212362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3661.114482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3661.118992] Lustre: Skipped 1 previous similar message [ 3661.121047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3661.155610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3661.792377] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3665.531598] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3666.930328] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3667.006665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3668.448671] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3668.534285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3671.521990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3671.525029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3671.526466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3671.530628] LustreError: Skipped 1 previous similar message [ 3671.532961] Lustre: Skipped 6 previous similar messages [ 3675.288431] Lustre: server umount lustre-OST0000 complete [ 3682.715694] Lustre: server umount lustre-MDT0000 complete [ 3683.956518] LustreError: 212361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811442 with bad export cookie 9558346521812843762 [ 3683.959521] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3683.960039] LustreError: 212361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3684.075144] Lustre: server umount lustre-MDT0001 complete [ 3685.288867] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:04:03 (1788811443) [ 3690.508210] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 3693.694386] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3705.338463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3705.419703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3705.430129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3705.432648] Lustre: Skipped 3 previous similar messages [ 3705.464811] Lustre: lustre-MDT0000: new disk, initializing [ 3705.466312] Lustre: Skipped 2 previous similar messages [ 3705.492686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3706.869260] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3710.933714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.973220] Lustre: 216961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3711.067456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3712.382139] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3715.846947] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3717.678381] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3719.855290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3719.975654] Lustre: 218927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3721.825271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3722.037819] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3725.952362] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3727.535119] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3727.610062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3729.031843] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3729.104675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3731.938878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3731.941274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3731.941456] LustreError: Skipped 1 previous similar message [ 3731.944087] Lustre: Skipped 5 previous similar messages [ 3744.224082] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3744.271926] Lustre: server umount lustre-OST0000 complete [ 3746.784817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3746.789913] Lustre: Skipped 5 previous similar messages [ 3751.576711] Lustre: server umount lustre-MDT0000 complete [ 3752.418093] LustreError: 216972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.424656] LustreError: 216972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3752.726208] LustreError: 216953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811511 with bad export cookie 9558346521812844644 [ 3752.729314] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3752.729315] LustreError: 216953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3752.837328] Lustre: server umount lustre-MDT0001 complete [ 3755.475153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.981674] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3759.747994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.109648] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3763.306822] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3765.129753] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3767.382021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3769.369412] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3771.700649] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3794.651700] Lustre: server umount lustre-OST0000 complete [ 3796.449288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3796.451311] Lustre: Skipped 12 previous similar messages [ 3810.272173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3810.336681] Lustre: server umount lustre-MDT0000 complete [ 3811.575626] LustreError: 221133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811570 with bad export cookie 9558346521812845526 [ 3811.578942] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3811.581127] LustreError: 221133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3811.700366] Lustre: server umount lustre-MDT0001 complete [ 3903.462103] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3904.472621] Key type lgssc unregistered [ 3904.607468] LNet: 224834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3904.610079] LNetError: 224834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3904.618284] LNet: Removed LNI 192.168.203.108@tcp [ 3904.939149] Key type .llcrypt unregistered [ 3904.940743] Key type ._llcrypt unregistered [ 3913.015580] Key type ._llcrypt registered [ 3913.016750] Key type .llcrypt registered [ 3913.051124] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 3919.325136] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3919.733417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3919.746807] alg: No test for adler32 (adler32-zlib) [ 3920.612210] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 3920.708494] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3922.288286] Key type lgssc registered [ 3922.679404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3934.203452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3934.208928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3935.291354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3935.303061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3935.338870] Lustre: lustre-MDT0000: new disk, initializing [ 3935.362430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3935.368846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3936.721636] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3940.834490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.883516] Lustre: 228152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3940.913432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3940.917026] Lustre: Skipped 1 previous similar message [ 3940.961939] Lustre: lustre-MDT0001: new disk, initializing [ 3940.998366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3941.011845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3941.015907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3942.373673] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3945.970693] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3947.859985] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3950.240854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3950.375331] Lustre: lustre-OST0000: new disk, initializing [ 3950.378063] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3950.382320] Lustre: 230120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3950.415853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3950.971774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3950.974865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3951.011329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3952.413268] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3956.064596] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3957.425200] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3957.489617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3958.916762] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3958.991558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3961.316966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3961.317352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.318345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3961.320615] LustreError: Skipped 1 previous similar message [ 3961.331055] Lustre: Skipped 1 previous similar message [ 3965.711814] Lustre: server umount lustre-OST0000 complete [ 3971.554094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.554859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3971.558445] Lustre: Skipped 1 previous similar message [ 3971.563072] Lustre: Skipped 2 previous similar messages [ 3976.673103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3976.676223] Lustre: Skipped 1 previous similar message [ 3981.280095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3981.355482] Lustre: server umount lustre-MDT0000 complete [ 3982.619375] LustreError: 228145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811741 with bad export cookie 15314509658167058148 [ 3982.622852] LustreError: 228145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3982.623457] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3982.753139] Lustre: server umount lustre-MDT0001 complete [ 3988.858676] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3992.340216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3992.472096] LustreError: 233051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3992.478911] LustreError: 233051:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3992.483443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3993.817617] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3996.655209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3996.783302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3998.120856] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3999.057370] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4001.191881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4003.243073] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4006.062538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4006.139351] Lustre: lustre-OST0001: new disk, initializing [ 4006.141114] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4006.143624] Lustre: 235357:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4006.184518] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4006.186867] Lustre: Skipped 1 previous similar message [ 4008.215290] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4008.708813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4008.712212] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4008.746696] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4013.921876] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4016.666362] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4020.293944] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:09:38 (1788811778) [ 4022.240829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4022.243503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4022.248055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4024.291175] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.295484] Lustre: Skipped 2 previous similar messages [ 4027.162484] Lustre: server umount lustre-MDT0000 complete [ 4028.359301] LustreError: 233033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811787 with bad export cookie 15314509658167059016 [ 4028.363594] LustreError: 233033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4028.364181] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4028.469376] Lustre: server umount lustre-MDT0001 complete [ 4039.938779] Lustre: server umount lustre-OST0000 complete [ 4051.486860] Lustre: server umount lustre-OST0001 complete [ 4053.515355] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 4056.760961] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 4069.960686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.059318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4070.070208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4070.112241] Lustre: lustre-MDT0000: new disk, initializing [ 4070.134480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.140919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4071.568976] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4076.043447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4076.085715] Lustre: 239295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4076.189179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4077.493360] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4081.284731] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4083.055443] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4085.584315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4085.706499] Lustre: lustre-OST0000: new disk, initializing [ 4085.708461] Lustre: Skipped 1 previous similar message [ 4085.710574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4085.712915] Lustre: Skipped 2 previous similar messages [ 4085.715335] Lustre: 241264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4087.482077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4087.485137] Lustre: Skipped 1 previous similar message [ 4087.487271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4087.521324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4087.879624] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4091.718378] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4093.160783] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4093.238927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4094.695070] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4094.764333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4098.022060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4098.022900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.035594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4098.039271] Lustre: Skipped 5 previous similar messages [ 4101.555172] Lustre: server umount lustre-OST0000 complete [ 4108.257493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4108.260628] Lustre: Skipped 3 previous similar messages [ 4109.300915] Lustre: server umount lustre-MDT0000 complete [ 4110.582444] LustreError: 239289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788811869 with bad export cookie 15314509658167061109 [ 4110.585392] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4110.586175] LustreError: 239289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4110.703675] Lustre: server umount lustre-MDT0001 complete [ 4113.517133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.671737] LustreError: 243479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.693695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.695939] Lustre: Skipped 2 previous similar messages [ 4115.075205] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4118.760347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4120.710569] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4123.835785] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4125.786205] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4128.544167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4131.168378] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4133.676131] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4281.314233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4281.315332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.316539] LustreError: Skipped 1 previous similar message [ 4281.319829] Lustre: Skipped 3 previous similar messages [ 4281.320390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4281.324072] Lustre: Skipped 1 previous similar message [ 4284.253272] Lustre: server umount lustre-OST0000 complete [ 4300.256110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4300.331240] Lustre: server umount lustre-MDT0000 complete [ 4301.616547] LustreError: 243460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788812060 with bad export cookie 15314509658167061991 [ 4301.622587] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4301.622672] LustreError: 243460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4301.763697] Lustre: server umount lustre-MDT0001 complete [ 4404.183425] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 4406.939797] Key type lgssc unregistered [ 4407.364897] LNet: 247082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4407.369910] LNetError: 247082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4407.382932] LNet: Removed LNI 192.168.203.108@tcp [ 4408.267260] Key type .llcrypt unregistered [ 4408.270534] Key type ._llcrypt unregistered [ 4425.468878] Key type ._llcrypt registered [ 4425.473140] Key type .llcrypt registered [ 4425.559064] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 4434.132934] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 4434.614214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4434.626402] alg: No test for adler32 (adler32-zlib) [ 4435.520411] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 4435.641171] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 4437.232211] Key type lgssc registered [ 4437.839398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4461.459948] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 4467.509556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4467.546100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4468.679601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4468.697510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4468.754309] Lustre: lustre-MDT0000: new disk, initializing [ 4468.800133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4468.811078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4471.477250] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4478.045149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4478.117376] Lustre: 251125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4478.153722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4478.156531] Lustre: Skipped 1 previous similar message [ 4478.198048] Lustre: lustre-MDT0001: new disk, initializing [ 4478.248462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4478.271800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4478.281419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4480.832910] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4484.004274] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4487.668259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4487.869492] Lustre: lustre-OST0000: new disk, initializing [ 4487.871788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4487.875846] Lustre: 252755:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4487.927652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4491.356822] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4495.367201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4495.373501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4495.430948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4497.364622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4497.500106] Lustre: lustre-OST0001: new disk, initializing [ 4497.503932] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4497.509699] Lustre: 253628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4497.583459] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4501.131631] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4505.110473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4505.118738] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4505.179293] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4508.123600] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4510.737362] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4519.393917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4519.398934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.407850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4520.421864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4520.426987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4520.434600] Lustre: Skipped 1 previous similar message [ 4520.443438] Lustre: Skipped 3 previous similar messages [ 4525.537322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4525.542609] Lustre: Skipped 3 previous similar messages [ 4530.661781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4530.666422] Lustre: Skipped 3 previous similar messages [ 4531.168131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4531.309445] Lustre: server umount lustre-MDT0000 complete [ 4533.549485] LustreError: 251115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788812292 with bad export cookie 13276871387794834531 [ 4533.555083] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4533.557815] LustreError: 251115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4533.756870] Lustre: server umount lustre-MDT0001 complete [ 4546.787741] Lustre: server umount lustre-OST0000 complete [ 4559.356650] Lustre: server umount lustre-OST0001 complete [ 4567.685148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4573.502613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4578.418619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4583.633303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4594.855603] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 4600.402581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.418184] Lustre: lustre-MDT0000: reset Object Index mappings [ 4600.687359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4602.937786] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4605.926928] LustreError: 258629:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.961748] LustreError: 258629:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4608.281258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.294178] Lustre: lustre-MDT0001: reset Object Index mappings [ 4611.033457] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4612.802753] Lustre: 259771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4616.691393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4616.978545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4616.984356] Lustre: Skipped 1 previous similar message [ 4618.028255] LustreError: 260125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.928830] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4623.332148] LustreError: 260229:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.399404] LustreError: 260123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.413169] LustreError: 260123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4625.420535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4626.794691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4630.752317] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4632.063646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4636.002313] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4638.400497] Lustre: 261639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4642.279717] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4642.284176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4642.288727] Lustre: Skipped 3 previous similar messages [ 4642.296909] Lustre: Skipped 1 previous similar message [ 4647.311429] Lustre: server umount lustre-MDT0000 complete [ 4647.399599] LustreError: 258628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4647.417971] LustreError: 258628:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4649.671677] LustreError: 258610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788812408 with bad export cookie 13276871387794841321 [ 4649.676370] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4649.686688] LustreError: 258610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4649.930316] Lustre: server umount lustre-MDT0001 complete [ 4662.798919] Lustre: server umount lustre-OST0000 complete [ 4675.473529] Lustre: server umount lustre-OST0001 complete [ 4687.786243] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:20:45 (1788812445) [ 4697.278764] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 4703.673033] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 4727.477852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.637805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4727.664723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4727.715704] Lustre: lustre-MDT0000: new disk, initializing [ 4727.788133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4727.797293] Lustre: Skipped 1 previous similar message [ 4727.811898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4730.570649] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4737.766677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.828684] Lustre: 265891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4737.870555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4737.873962] Lustre: Skipped 1 previous similar message [ 4737.930588] Lustre: lustre-MDT0001: new disk, initializing [ 4738.022732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4738.030595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4740.748827] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4747.133508] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4750.867489] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4755.333166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4755.537794] Lustre: lustre-OST0000: new disk, initializing [ 4755.542514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4755.553039] Lustre: 267857:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4756.898952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4756.910398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4757.030466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4759.726920] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4766.018886] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4768.910873] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4769.070397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4771.708814] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4771.823839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4777.446083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4777.446341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.451919] LustreError: Skipped 1 previous similar message [ 4777.458469] Lustre: Skipped 1 previous similar message [ 4777.459871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4777.469010] Lustre: Skipped 3 previous similar messages [ 4787.171139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4787.293779] Lustre: server umount lustre-OST0000 complete [ 4792.801384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4792.813692] Lustre: Skipped 2 previous similar messages [ 4796.084233] Lustre: server umount lustre-MDT0000 complete [ 4797.921724] LustreError: 265902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.931309] LustreError: 265902:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4798.683725] LustreError: 265881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788812557 with bad export cookie 13276871387794843407 [ 4798.695129] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4798.696889] LustreError: 265881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4798.897347] Lustre: server umount lustre-MDT0001 complete [ 4811.006894] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 4817.603382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4817.892196] LustreError: 270786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4817.936808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.941984] Lustre: Skipped 2 previous similar messages [ 4821.112598] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4827.227056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4830.266222] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4832.095661] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4836.362182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4840.639590] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4845.930488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4846.060932] Lustre: lustre-OST0001: new disk, initializing [ 4846.065357] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4846.072043] Lustre: 273094:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4847.992074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4847.997871] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4848.056068] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4849.748565] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4855.637113] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4857.462230] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4860.293730] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4860.343783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4867.019302] Lustre: server umount lustre-MDT0000 complete [ 4868.579781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4868.581664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4868.582043] LustreError: 271964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4868.582053] LustreError: 271964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4868.587130] Lustre: Skipped 3 previous similar messages [ 4868.870599] LustreError: 270767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788812627 with bad export cookie 13276871387794844289 [ 4868.874803] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4868.875702] LustreError: 270767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4871.095281] Lustre: server umount lustre-MDT0001 complete [ 4883.999875] Lustre: server umount lustre-OST0000 complete [ 4895.184753] Lustre: server umount lustre-OST0001 complete [ 4904.114631] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 4909.557509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.844326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4909.847383] Lustre: Skipped 3 previous similar messages [ 4911.814228] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4916.055307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4918.787705] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4920.452466] Lustre: 277575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4924.248049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4928.000211] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4932.549606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4934.775630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4934.809530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4936.412964] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4940.818426] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4942.829273] Lustre: 279450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4945.460218] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4945.478734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4952.654642] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:25:10 (1788812710) [ 4965.862559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.864222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4965.875757] Lustre: Skipped 2 previous similar messages [ 4965.880750] Lustre: Skipped 7 previous similar messages [ 4967.368711] Lustre: server umount lustre-MDT0000 complete [ 4969.430253] LustreError: 276414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788812728 with bad export cookie 13276871387794850659 [ 4969.432134] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4969.438170] LustreError: 276414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4970.977588] LustreError: 278919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4970.995375] LustreError: 278919:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4971.674763] Lustre: server umount lustre-MDT0001 complete [ 4988.384134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4988.443631] Lustre: server umount lustre-OST0000 complete [ 4996.812556] Lustre: server umount lustre-OST0001 complete [ 5001.207924] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 5007.536679] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 5035.038576] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 5041.064935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5041.217187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5041.240097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5041.292678] Lustre: lustre-MDT0000: new disk, initializing [ 5041.345477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5041.352267] Lustre: Skipped 3 previous similar messages [ 5041.366748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5044.162634] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5050.891841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5050.955825] Lustre: 283974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5051.132362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5053.360525] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5059.966907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5060.121432] Lustre: 285607:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5061.272908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5063.177099] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5069.291151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5069.378492] Lustre: 286476:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5071.417677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5072.919647] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5079.975207] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5083.153585] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5083.159876] Lustre: Skipped 1 previous similar message [ 5091.834674] hrtimer: interrupt took 9971510 ns [ 5100.027942] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:27:37 (1788812857) [ 5102.562627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5102.576753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5102.607750] Lustre: Skipped 3 previous similar messages [ 5102.621912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5102.637529] Lustre: Skipped 2 previous similar messages [ 5104.842533] Lustre: server umount lustre-MDT0000 complete [ 5107.205861] LustreError: 283985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5107.254479] LustreError: 283985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5109.792697] LustreError: 284768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788812868 with bad export cookie 13276871387794858597 [ 5109.833846] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5109.844429] LustreError: 284768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5110.315381] Lustre: server umount lustre-MDT0001 complete [ 5125.334676] Lustre: server umount lustre-OST0000 complete [ 5128.162844] Lustre: 248631:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788812871/real 1788812871] req@ffff9f67ca70ad80 x1875705535466368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788812887 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5130.570812] Lustre: server umount lustre-OST0001 complete [ 5140.016572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5145.760714] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5154.080923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5158.461408] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5164.818716] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5169.656876] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5175.311913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5180.479848] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5186.437865] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5198.596931] loop1: detected capacity change from 0 to 204800000 [ 5198.833886] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5198.844239] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5198.859581] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5202.523515] loop1: detected capacity change from 0 to 204800000 [ 5206.799672] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5206.873425] LustreError: 288708:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs25190: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5206.893750] LustreError: 288708:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs25190 [ 5206.900369] LustreError: 288708:0:(mgs_handler.c:517:mgs_target_reg()) fs25190-MDT0001: Error checking target index: rc = -13 [ 5206.906490] LustreError: fs25190-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5206.911683] LustreError: 292438:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 5206.916318] LustreError: 292438:0:(tgt_mount.c:2128:server_put_super()) no obd fs25190-MDT0001 [ 5206.920150] LustreError: 292438:0:(tgt_mount.c:116:server_deregister_mount()) fs25190-MDT0001 not registered [ 5206.944243] Lustre: server umount fs25190-MDT0001 complete [ 5206.948506] LustreError: 292438:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5213.887620] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.949210] Lustre: Setting parameter fs25190-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs25190-MDT0001 [ 5213.989151] Lustre: srv-fs25190-MDT0001: No data found on store. Initialize space. [ 5213.992849] Lustre: Skipped 4 previous similar messages [ 5214.071648] Lustre: fs25190-MDT0001: new disk, initializing [ 5214.074984] Lustre: Skipped 3 previous similar messages [ 5214.186448] Lustre: srv-fs25190-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5214.193863] Lustre: 292904:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25190-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5214.200659] LustreError: 292904:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25190-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5217.546580] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5222.975774] Lustre: server umount fs25190-MDT0001 complete [ 5233.420665] loop1: detected capacity change from 0 to 204800000 [ 5238.062854] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.336098] Lustre: 294470:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25190-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5238.347516] LustreError: 294470:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25190-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5241.666609] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5245.387918] Lustre: server umount fs25190-MDT0001 complete [ 5252.579571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5252.588253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5252.602372] Lustre: Skipped 6 previous similar messages [ 5252.607672] Lustre: Skipped 1 previous similar message [ 5258.538670] Lustre: server umount lustre-MDT0000 complete [ 5262.823043] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 5265.336133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.471844] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5269.077238] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5271.029235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5275.217458] loop1: detected capacity change from 0 to 204800000 [ 5279.638325] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5279.954100] Lustre: 296915:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25190-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5279.958210] LustreError: 296915:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25190-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5283.339130] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5286.936600] Lustre: server umount fs25190-MDT0001 complete [ 5317.099683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5329.888452] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5330.170748] Lustre: server umount lustre-OST0000 complete [ 5337.573364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5349.349857] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5349.641665] Lustre: server umount lustre-MDT0000 complete [ 5353.562368] LustreError: 288699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788813112 with bad export cookie 13276871387794861278 [ 5353.565751] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5353.577054] LustreError: 288699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5354.483917] Lustre: server umount lustre-MDT0001 complete [ 5365.782765] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 5368.476570] Key type lgssc unregistered [ 5368.788686] LNet: 299260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5368.797113] LNetError: 299260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5368.834283] LNet: Removed LNI 192.168.203.108@tcp [ 5369.625181] Key type .llcrypt unregistered [ 5369.629486] Key type ._llcrypt unregistered [ 5387.116392] Key type ._llcrypt registered [ 5387.120434] Key type .llcrypt registered [ 5387.220607] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 5388.562887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5388.586936] alg: No test for adler32 (adler32-zlib) [ 5389.805466] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 5390.179633] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 5391.881740] Key type lgssc registered [ 5393.103552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5400.996729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5401.026038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5402.526170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5407.401876] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5412.836307] LustreError: 300529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5412.858569] LustreError: 300529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5415.549648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5415.937110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5420.197282] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5427.416235] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5432.963991] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5439.837932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5440.281431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5441.328236] LustreError: 302358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5445.979215] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5446.631256] LustreError: 302358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.770196] LustreError: 302359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.796901] LustreError: 302359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5453.637944] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5457.908979] LustreError: 302359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5457.936907] LustreError: 302359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5468.137677] LustreError: 302360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.151972] LustreError: 302360:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5469.530475] loop1: detected capacity change from 0 to 204800000 [ 5469.746329] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5469.761496] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5469.772676] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5474.547443] loop1: detected capacity change from 0 to 204800000 [ 5480.471580] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5480.657900] Lustre: fs31245-OST0000: new disk, initializing [ 5480.664691] Lustre: srv-fs31245-OST0000: No data found on store. Initialize space. [ 5480.674622] Lustre: 304261:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5480.784587] Lustre: fs31245-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5486.104332] LustreError: 302980:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5486.132909] LustreError: 302980:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5487.289130] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5495.725727] Lustre: server umount fs31245-OST0000 complete [ 5507.843422] loop1: detected capacity change from 0 to 204800000 [ 5514.144641] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5514.349684] Lustre: fs31245-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.342055] LustreError: 302358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5519.368321] LustreError: 302358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 5520.583779] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5524.468315] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 5528.770548] loop2: detected capacity change from 0 to 204800000 [ 5528.975307] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5528.989109] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5528.997687] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5529.585941] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 5534.004189] loop2: detected capacity change from 0 to 204800000 [ 5539.313780] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5539.417105] LustreError: fs31245-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 5539.427538] LustreError: fs31245-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5539.435502] LustreError: 307299:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 5539.440993] LustreError: 307299:0:(tgt_mount.c:2128:server_put_super()) no obd fs31245-OST0001 [ 5539.445334] LustreError: 307299:0:(tgt_mount.c:116:server_deregister_mount()) fs31245-OST0001 not registered [ 5539.469201] Lustre: server umount fs31245-OST0001 complete [ 5539.472244] LustreError: 307299:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5549.028449] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5549.300802] Lustre: fs31245-OST0001: new disk, initializing [ 5549.320678] Lustre: srv-fs31245-OST0001: No data found on store. Initialize space. [ 5549.328570] Lustre: 307860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5549.410933] Lustre: fs31245-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5556.740541] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5564.901160] Lustre: server umount fs31245-OST0000 complete [ 5573.567320] Lustre: server umount fs31245-OST0001 complete [ 5583.382385] LustreError: 302980:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.416985] LustreError: 302980:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 36 previous similar messages [ 5586.243405] loop1: detected capacity change from 0 to 204800000 [ 5592.063534] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5592.315851] Lustre: fs31245-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.029465] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5605.677673] loop2: detected capacity change from 0 to 204800000 [ 5610.989829] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5618.319399] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5627.886318] Lustre: server umount fs31245-OST0000 complete [ 5637.893624] Lustre: server umount fs31245-OST0001 complete [ 5652.452443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.456689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5652.457179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5652.494478] Lustre: Skipped 1 previous similar message [ 5652.537292] Lustre: Skipped 1 previous similar message [ 5657.574745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5657.582215] Lustre: Skipped 1 previous similar message [ 5658.476247] Lustre: server umount lustre-OST0000 complete [ 5662.693573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.719636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5662.730884] Lustre: Skipped 1 previous similar message [ 5667.811149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5667.829541] Lustre: Skipped 1 previous similar message [ 5668.988329] Lustre: server umount lustre-MDT0000 complete [ 5673.050912] LustreError: 300509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788813432 with bad export cookie 17931924333950829311 [ 5673.066095] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5673.067976] LustreError: 300509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5673.475440] Lustre: server umount lustre-MDT0001 complete [ 5686.085681] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 5689.086360] Key type lgssc unregistered [ 5689.411164] LNet: 313784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5689.420105] LNetError: 313784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5689.436166] LNet: Removed LNI 192.168.203.108@tcp [ 5690.343152] Key type .llcrypt unregistered [ 5690.347370] Key type ._llcrypt unregistered [ 5722.270793] Key type ._llcrypt registered [ 5722.273523] Key type .llcrypt registered [ 5722.374201] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 5738.540207] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 5739.660819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5739.734105] alg: No test for adler32 (adler32-zlib) [ 5740.882478] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 5741.109979] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 5742.768213] Key type lgssc registered [ 5744.096608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5783.130862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5783.165321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5784.494357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5784.521596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5784.625295] Lustre: lustre-MDT0000: new disk, initializing [ 5784.730990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5784.772740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5789.330309] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5799.856639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5800.028904] Lustre: 317665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5800.075386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5800.078916] Lustre: Skipped 1 previous similar message [ 5800.161791] Lustre: lustre-MDT0001: new disk, initializing [ 5800.244236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5800.284659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5800.294802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5804.767891] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5814.473761] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5820.724322] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5828.339184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5828.841350] Lustre: lustre-OST0000: new disk, initializing [ 5828.854921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5828.863732] Lustre: 319634:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5829.027897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5836.537412] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5837.994361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5838.026958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5838.276898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5849.251761] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5851.746976] Lustre: Mounted lustre-client - version 2.17.58_39_g569731d [ 5853.115944] Lustre: Unmounted lustre-client [ 5863.906611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5863.906897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.916858] LustreError: Skipped 1 previous similar message [ 5863.920098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5863.930103] Lustre: Skipped 1 previous similar message [ 5863.953765] Lustre: Skipped 1 previous similar message [ 5866.980689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5866.993966] Lustre: Skipped 1 previous similar message [ 5868.610717] Lustre: server umount lustre-OST0000 complete [ 5877.223074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5877.223323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5877.239938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5877.252171] Lustre: Skipped 1 previous similar message [ 5878.647075] Lustre: server umount lustre-MDT0000 complete [ 5882.262547] LustreError: 318461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788813641 with bad export cookie 14576001748063096462 [ 5882.274532] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5882.279129] LustreError: 318461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5882.599937] Lustre: server umount lustre-MDT0001 complete [ 5893.441601] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 5896.211766] Key type lgssc unregistered [ 5910.112746] LNet: 1 local NIs in recovery (showing 1): 192.168.203.108@tcp [ 5910.881085] LNet: 321796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5910.896039] LNetError: 321796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5981.612372] LNet: Removed LNI 192.168.203.108@tcp [ 5982.542968] Key type .llcrypt unregistered [ 5982.547238] Key type ._llcrypt unregistered [ 5999.597365] Key type ._llcrypt registered [ 5999.606091] Key type .llcrypt registered [ 5999.740533] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 6016.225980] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6017.711424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6017.732241] alg: No test for adler32 (adler32-zlib) [ 6018.861841] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 6019.094841] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 6020.784272] Key type lgssc registered [ 6021.804240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6055.964959] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:43:33 (1788813813) [ 6064.453934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6064.472677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6065.737866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6065.776760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6065.875159] Lustre: lustre-MDT0000: new disk, initializing [ 6065.939777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6065.983076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6070.924591] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6082.298765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6082.526600] Lustre: 325016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6082.634181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6082.646416] Lustre: Skipped 1 previous similar message [ 6082.773260] Lustre: lustre-MDT0001: new disk, initializing [ 6082.904671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6082.975237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6083.000271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6087.587995] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6098.255528] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6104.591845] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6111.614403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6111.882571] Lustre: lustre-OST0000: new disk, initializing [ 6111.886105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6111.891798] Lustre: 326983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6111.985421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6118.444839] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6120.019260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6120.036945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6120.159286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6126.490425] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6140.897672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6140.912176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.931338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6140.936313] Lustre: Skipped 1 previous similar message [ 6146.018814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6146.031126] Lustre: Skipped 1 previous similar message [ 6146.834343] Lustre: server umount lustre-OST0000 complete [ 6151.142504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.145779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6151.164992] Lustre: Skipped 2 previous similar messages [ 6151.178030] Lustre: Skipped 1 previous similar message [ 6156.258283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6156.267466] Lustre: Skipped 1 previous similar message [ 6157.181182] Lustre: server umount lustre-MDT0000 complete [ 6160.940890] LustreError: 325009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788813919 with bad export cookie 3283713729975099016 [ 6160.943338] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6160.951891] LustreError: 325009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6161.232942] Lustre: server umount lustre-MDT0001 complete [ 6171.847905] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6174.605284] Key type lgssc unregistered [ 6174.862203] LNet: 328890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6174.870299] LNetError: 328890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6174.881994] LNet: Removed LNI 192.168.203.108@tcp [ 6175.709530] Key type .llcrypt unregistered [ 6175.713303] Key type ._llcrypt unregistered [ 6195.735099] Key type ._llcrypt registered [ 6195.738304] Key type .llcrypt registered [ 6195.831160] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6196.926049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6196.960888] alg: No test for adler32 (adler32-zlib) [ 6198.028468] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 6198.285841] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 6199.936339] Key type lgssc registered [ 6201.027950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6207.907581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6207.927426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6209.357165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6213.430391] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6219.755873] LustreError: 330164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6219.774564] LustreError: 330164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6221.544717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6221.941375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6226.317880] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6233.575301] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6238.733908] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6245.167871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6245.512203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6251.213154] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6258.093633] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6267.561387] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6268.866965] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6272.995918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6273.008073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6273.024533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6275.563462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6275.576975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.116870] Lustre: server umount lustre-MDT0000 complete [ 6280.676626] LustreError: 330165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6280.700613] LustreError: 330165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6283.147035] LustreError: 330145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814042 with bad export cookie 13385160674755826255 [ 6283.151848] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6283.160256] LustreError: 330145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6285.795436] LustreError: 330161:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6285.795734] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6285.817894] LustreError: 330161:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6285.864662] Lustre: Skipped 1 previous similar message [ 6285.879417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6285.893848] Lustre: Skipped 3 previous similar messages [ 6288.801045] LustreError: 330159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6288.804576] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6288.826951] LustreError: 330159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6288.846354] Lustre: Skipped 1 previous similar message [ 6289.507221] Lustre: server umount lustre-MDT0001 complete [ 6293.344167] Lustre: server umount lustre-OST0000 complete [ 6301.508418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6305.710581] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6308.305836] Lustre: 334344:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.108@tcp [ 6310.223698] Lustre: server umount MGS complete [ 6318.550971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.956154] LustreError: 334915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6319.013102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.716521] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6328.289469] LustreError: 334916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6328.307787] LustreError: 334916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6330.625768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.051034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6335.423481] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6343.311897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6343.626361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6349.866444] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6356.636381] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6358.045237] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 6363.137125] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6364.618500] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6370.273908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6370.285811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6370.299963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6371.298025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6371.307540] LustreError: Skipped 1 previous similar message [ 6374.999713] Lustre: server umount lustre-OST0000 complete [ 6381.538737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.562839] Lustre: Skipped 1 previous similar message [ 6381.574323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6381.586174] Lustre: Skipped 3 previous similar messages [ 6384.620995] Lustre: server umount lustre-MDT0000 complete [ 6386.662467] LustreError: 334910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.683347] LustreError: 334910:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6387.928762] LustreError: 336373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814146 with bad export cookie 13385160674755827095 [ 6387.933370] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6387.936518] LustreError: 336373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6388.216125] Lustre: server umount lustre-MDT0001 complete [ 6397.351542] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6399.955135] Key type lgssc unregistered [ 6400.235758] LNet: 338350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6400.244266] LNetError: 338350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6400.268173] LNet: Removed LNI 192.168.203.108@tcp [ 6401.044249] Key type .llcrypt unregistered [ 6401.047132] Key type ._llcrypt unregistered [ 6418.696494] Key type ._llcrypt registered [ 6418.698324] Key type .llcrypt registered [ 6418.837509] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 6433.289260] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6434.032801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6434.160667] alg: No test for adler32 (adler32-zlib) [ 6435.157851] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 6435.354705] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 6437.048212] Key type lgssc registered [ 6437.942752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6470.242085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6470.288349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6471.585440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6471.627290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6471.735929] Lustre: lustre-MDT0000: new disk, initializing [ 6471.821383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6471.844670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6476.087675] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6485.575829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.693545] Lustre: 341505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6485.732670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6485.735830] Lustre: Skipped 1 previous similar message [ 6485.812152] Lustre: lustre-MDT0001: new disk, initializing [ 6485.916889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6485.955484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6485.965262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6489.681811] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6498.384194] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6503.404232] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6508.860918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6509.095409] Lustre: lustre-OST0000: new disk, initializing [ 6509.101106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6509.108319] Lustre: 343476:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6509.181787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6513.905000] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6519.330331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6519.342633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6519.460535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6521.686236] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6525.342710] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6525.555963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6529.336432] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6529.592824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6534.631260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6534.647704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6534.665950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6535.147640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.167437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6535.169437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6535.187265] Lustre: Skipped 1 previous similar message [ 6537.477539] Lustre: server umount lustre-OST0000 complete [ 6542.309922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6542.317452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6542.339558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6545.378087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6545.379433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6545.402196] Lustre: Skipped 1 previous similar message [ 6546.907108] Lustre: server umount lustre-MDT0000 complete [ 6550.087270] LustreError: 341498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814309 with bad export cookie 7232369392121121363 [ 6550.088104] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6550.100153] LustreError: 341498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6550.388803] Lustre: server umount lustre-MDT0001 complete [ 6558.067390] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:51:55 (1788814315) [ 6571.050301] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 6579.751257] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6606.744569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6606.903199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6606.933902] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6606.985541] Lustre: lustre-MDT0000: new disk, initializing [ 6607.050972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6607.062292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6610.258134] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6618.241991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6618.356925] Lustre: 348232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6618.394854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6618.397673] Lustre: Skipped 1 previous similar message [ 6618.467708] Lustre: lustre-MDT0001: new disk, initializing [ 6618.527883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6618.558334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6618.573775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6622.546995] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6630.299931] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6634.445902] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6640.053251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6640.324759] Lustre: lustre-OST0000: new disk, initializing [ 6640.328964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6640.343746] Lustre: 350198:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6640.458481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6645.446544] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6647.852438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6647.861819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6647.978074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6653.569658] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6678.498073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6678.506512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6678.518086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6682.699802] Lustre: server umount lustre-OST0000 complete [ 6685.155647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6685.164292] LustreError: Skipped 1 previous similar message [ 6689.253090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6689.255060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6689.262442] Lustre: Skipped 2 previous similar messages [ 6689.272801] Lustre: Skipped 4 previous similar messages [ 6691.367190] Lustre: server umount lustre-MDT0000 complete [ 6694.203243] LustreError: 348223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814453 with bad export cookie 7232369392121122245 [ 6694.209950] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6694.218136] LustreError: 348223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6694.450826] Lustre: server umount lustre-MDT0001 complete [ 6701.876432] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6704.026517] Key type lgssc unregistered [ 6704.269099] LNet: 352346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6704.277241] LNetError: 352346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6704.294701] LNet: Removed LNI 192.168.203.108@tcp [ 6704.970498] Key type .llcrypt unregistered [ 6704.972206] Key type ._llcrypt unregistered [ 6769.846401] Key type ._llcrypt registered [ 6769.849562] Key type .llcrypt registered [ 6769.946282] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6770.174495] Key type .llcrypt unregistered [ 6770.177073] Key type ._llcrypt unregistered [ 6821.652607] Key type ._llcrypt registered [ 6821.654631] Key type .llcrypt registered [ 6821.720962] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6821.906136] Key type .llcrypt unregistered [ 6821.910625] Key type ._llcrypt unregistered [ 6843.245795] Key type ._llcrypt registered [ 6843.248963] Key type .llcrypt registered [ 6843.337592] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6843.604575] Key type .llcrypt unregistered [ 6843.607232] Key type ._llcrypt unregistered [ 6863.549232] Key type ._llcrypt registered [ 6863.551708] Key type .llcrypt registered [ 6863.644783] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6863.869394] Key type .llcrypt unregistered [ 6863.871995] Key type ._llcrypt unregistered [ 6909.374162] Key type ._llcrypt registered [ 6909.377526] Key type .llcrypt registered [ 6909.447225] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6909.633749] Key type .llcrypt unregistered [ 6909.635692] Key type ._llcrypt unregistered [ 6928.142632] Key type ._llcrypt registered [ 6928.146048] Key type .llcrypt registered [ 6928.220159] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6928.423348] Key type .llcrypt unregistered [ 6928.428583] Key type ._llcrypt unregistered [ 6945.835636] Key type ._llcrypt registered [ 6945.838552] Key type .llcrypt registered [ 6945.908491] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6946.101457] Key type .llcrypt unregistered [ 6946.104579] Key type ._llcrypt unregistered [ 6964.021992] Key type ._llcrypt registered [ 6964.024516] Key type .llcrypt registered [ 6964.100295] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6964.277338] Key type .llcrypt unregistered [ 6964.279274] Key type ._llcrypt unregistered [ 6982.313045] Key type ._llcrypt registered [ 6982.316233] Key type .llcrypt registered [ 6982.387949] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6982.578597] Key type .llcrypt unregistered [ 6982.585837] Key type ._llcrypt unregistered [ 6999.731313] Key type ._llcrypt registered [ 6999.733884] Key type .llcrypt registered [ 6999.812101] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6999.983317] Key type .llcrypt unregistered [ 6999.986094] Key type ._llcrypt unregistered [ 7037.262302] Key type ._llcrypt registered [ 7037.264666] Key type .llcrypt registered [ 7037.329791] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 7037.529387] Key type .llcrypt unregistered [ 7037.532422] Key type ._llcrypt unregistered [ 7054.717590] Key type ._llcrypt registered [ 7054.719445] Key type .llcrypt registered [ 7054.775649] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 7054.938265] Key type .llcrypt unregistered [ 7054.941964] Key type ._llcrypt unregistered [ 7070.855265] Key type ._llcrypt registered [ 7070.856680] Key type .llcrypt registered [ 7070.925283] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 7071.111443] Key type .llcrypt unregistered [ 7071.113307] Key type ._llcrypt unregistered [ 7100.624321] Key type ._llcrypt registered [ 7100.626287] Key type .llcrypt registered [ 7100.687081] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 7110.273549] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7110.737498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7110.758374] alg: No test for adler32 (adler32-zlib) [ 7111.700678] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 7111.856569] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 7113.488235] Key type lgssc registered [ 7114.264951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7133.532542] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7133.558913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.689765] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7134.716862] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7134.776506] Lustre: tst250fs-MDT0000: new disk, initializing [ 7134.837643] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7134.856702] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7137.367894] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7143.857532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.931587] Lustre: 372249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7143.970266] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7143.974597] Lustre: Skipped 1 previous similar message [ 7144.022673] Lustre: tst250fs-MDT0001: new disk, initializing [ 7144.083324] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7144.110170] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7144.121537] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7146.699408] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7149.960198] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7154.081379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7154.275392] Lustre: tst250fs-OST0000: new disk, initializing [ 7154.279634] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7154.286120] Lustre: 373879:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7154.350810] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7157.201662] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7157.205995] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7157.275130] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7157.524091] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7163.296272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7163.387231] Lustre: tst250fs-OST0001: new disk, initializing [ 7163.390738] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7163.395206] Lustre: 374750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7163.461293] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7163.739621] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7163.747761] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7163.801666] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7166.527393] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7169.866688] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7184.358530] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7184.361653] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7184.370821] Lustre: Skipped 3 previous similar messages [ 7184.374090] Lustre: Skipped 2 previous similar messages [ 7189.473528] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7189.478896] Lustre: Skipped 3 previous similar messages [ 7194.596534] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7194.605468] Lustre: Skipped 3 previous similar messages [ 7197.152588] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7197.267384] Lustre: server umount tst250fs-MDT0000 complete [ 7199.469377] LustreError: 372241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814958 with bad export cookie 3699747620397194444 [ 7199.472860] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7199.477600] LustreError: 372241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7199.644784] Lustre: server umount tst250fs-MDT0001 complete [ 7211.796672] Lustre: server umount tst250fs-OST0000 complete [ 7223.132324] Lustre: server umount tst250fs-OST0001 complete [ 7233.584535] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 7238.215133] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7256.617992] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 17:03:34 (1788815014) [ 7257.548992] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7266.833258] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 7272.371110] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7289.091364] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7070 sec ========== 17:04:07 (1788815047) [ 7289.976449] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:04:08 (1788815048) === [ 7291.758339] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:04:09 (1788815049) === [ 7313.750974] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 7315.462316] Key type lgssc unregistered [ 7315.635451] LNet: 383863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7315.644166] LNetError: 383863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7315.658671] LNet: Removed LNI 192.168.203.108@tcp [ 7316.163357] Key type .llcrypt unregistered [ 7316.167491] Key type ._llcrypt unregistered